Introduction to Neural Networks - Rene wind Problem Statement¶¶

Problem Statement¶

Build various classification models, tune them and find the best one to identify failures so that the generators can be repaired before failing/breaking to reduce overall maintenance cost.

Nature of predictions:

  1. True Positives (TP) = Failures correctly predicted by model. Hence this will give repair cost.

  2. False Negatives (FN) = Failures not detected by model. Hence the dmaage will be done and it will give replacement cost.

  3. False Positives (FP) = Failures that are wrongly detected by model. Hence resulting in unnecessary inspection cost.

In the end, Inspection cost < Repair cost < Replacement cost.

Data Dictionary¶

Sensor data split into train and test csv files with 40 predictor variables and 1 target variable.

Target variable 1 - 'failure' Target variable 0 - 'no failure'

In [1]:
#Installing libraries
!pip install tensorflow==2.15.0

Collecting tensorflow==2.15.0
  Downloading tensorflow-2.15.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.4 kB)
Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (1.4.0)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (1.6.3)
Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (25.2.10)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (0.6.0)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (0.2.0)
Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (3.12.1)
Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (18.1.1)
Collecting ml-dtypes~=0.2.0 (from tensorflow==2.15.0)
  Downloading ml_dtypes-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (20 kB)
Requirement already satisfied: numpy<2.0.0,>=1.23.5 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (1.26.4)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (3.4.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (24.2)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (4.25.6)
Requirement already satisfied: setuptools in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (75.1.0)
Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (1.17.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (2.5.0)
Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (4.12.2)
Collecting wrapt<1.15,>=1.11.0 (from tensorflow==2.15.0)
  Downloading wrapt-1.14.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (0.37.1)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (1.70.0)
Collecting tensorboard<2.16,>=2.15 (from tensorflow==2.15.0)
  Downloading tensorboard-2.15.2-py3-none-any.whl.metadata (1.7 kB)
Collecting tensorflow-estimator<2.16,>=2.15.0 (from tensorflow==2.15.0)
  Downloading tensorflow_estimator-2.15.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting keras<2.16,>=2.15.0 (from tensorflow==2.15.0)
  Downloading keras-2.15.0-py3-none-any.whl.metadata (2.4 kB)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.11/dist-packages (from astunparse>=1.6.0->tensorflow==2.15.0) (0.45.1)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (2.27.0)
Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (1.2.1)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.7)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (2.32.3)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.1.3)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.11/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (5.5.1)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.11/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (0.4.1)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.11/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.11/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (2.0.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (2025.1.31)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.11/dist-packages (from werkzeug>=1.0.1->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.0.2)
Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.11/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (0.6.1)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.11/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.2.2)
Downloading tensorflow-2.15.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (475.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 475.3/475.3 MB 1.2 MB/s eta 0:00:00
Downloading keras-2.15.0-py3-none-any.whl (1.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 6.1 MB/s eta 0:00:00
Downloading ml_dtypes-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 8.7 MB/s eta 0:00:00
Downloading tensorboard-2.15.2-py3-none-any.whl (5.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 8.5 MB/s eta 0:00:00
Downloading tensorflow_estimator-2.15.0-py2.py3-none-any.whl (441 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 442.0/442.0 kB 12.3 MB/s eta 0:00:00
Downloading wrapt-1.14.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.4/78.4 kB 2.2 MB/s eta 0:00:00
Installing collected packages: wrapt, tensorflow-estimator, ml-dtypes, keras, tensorboard, tensorflow
  Attempting uninstall: wrapt
    Found existing installation: wrapt 1.17.2
    Uninstalling wrapt-1.17.2:
      Successfully uninstalled wrapt-1.17.2
  Attempting uninstall: ml-dtypes
    Found existing installation: ml-dtypes 0.4.1
    Uninstalling ml-dtypes-0.4.1:
      Successfully uninstalled ml-dtypes-0.4.1
  Attempting uninstall: keras
    Found existing installation: keras 3.8.0
    Uninstalling keras-3.8.0:
      Successfully uninstalled keras-3.8.0
  Attempting uninstall: tensorboard
    Found existing installation: tensorboard 2.18.0
    Uninstalling tensorboard-2.18.0:
      Successfully uninstalled tensorboard-2.18.0
  Attempting uninstall: tensorflow
    Found existing installation: tensorflow 2.18.0
    Uninstalling tensorflow-2.18.0:
      Successfully uninstalled tensorflow-2.18.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow-text 2.18.1 requires tensorflow<2.19,>=2.18.0, but you have tensorflow 2.15.0 which is incompatible.
tensorstore 0.1.71 requires ml_dtypes>=0.3.1, but you have ml-dtypes 0.2.0 which is incompatible.
tf-keras 2.18.0 requires tensorflow<2.19,>=2.18, but you have tensorflow 2.15.0 which is incompatible.
Successfully installed keras-2.15.0 ml-dtypes-0.2.0 tensorboard-2.15.2 tensorflow-2.15.0 tensorflow-estimator-2.15.0 wrapt-1.14.1

Importing Libraries¶

In [2]:
#Import necessary libraries for EDA
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

#Sklearn library imports
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, OneHotEncoder,StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.metrics import confusion_matrix, f1_score,accuracy_score, recall_score, precision_score, classification_report, precision_recall_curve

# Time related functions.
import time

#Imports the tensorflow,keras and layers.
import tensorflow
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Dense, Input, Dropout,BatchNormalization
from tensorflow.keras.initializers import HeNormal, GlorotUniform, RandomNormal
from tensorflow.keras.regularizers import l1_l2
from tensorflow.keras import backend

# to suppress unnecessary warnings
import warnings
warnings.filterwarnings("ignore")
In [3]:
# Set the seed using keras.utils.set_random_seed.
tf.keras.utils.set_random_seed(812)

# To reproduce the same output
tf.config.experimental.enable_op_determinism()

Data Overview¶

In [4]:
#Import the data from drive

from google.colab import drive
train_df = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/Intro_NN_Renewind/Train.csv')
test_df = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/Intro_NN_Renewind/Test.csv')
In [5]:
# Make a copy of the dataset
df_temp = train_df.copy()
In [6]:
#View a glimpse of the Training and Testing dataset

print(train_df.head())
print(test_df.head())

         V1        V2        V3        V4        V5        V6        V7  \
0 -4.464606 -4.679129  3.101546  0.506130 -0.221083 -2.032511 -2.910870   
1  3.365912  3.653381  0.909671 -1.367528  0.332016  2.358938  0.732600   
2 -3.831843 -5.824444  0.634031 -2.418815 -1.773827  1.016824 -2.098941   
3  1.618098  1.888342  7.046143 -1.147285  0.083080 -1.529780  0.207309   
4 -0.111440  3.872488 -3.758361 -2.982897  3.792714  0.544960  0.205433   

         V8        V9       V10  ...       V32       V33        V34       V35  \
0  0.050714 -1.522351  3.761892  ...  3.059700 -1.690440   2.846296  2.235198   
1 -4.332135  0.565695 -0.101080  ... -1.795474  3.032780  -2.467514  1.894599   
2 -3.173204 -2.081860  5.392621  ... -0.257101  0.803550   4.086219  2.292138   
3 -2.493629  0.344926  2.118578  ... -3.584425 -2.577474   1.363769  0.622714   
4  4.848994 -1.854920 -6.220023  ...  8.265896  6.629213 -10.068689  1.222987   

        V36       V37       V38       V39       V40  Target  
0  6.667486  0.443809 -2.369169  2.950578 -3.480324       0  
1 -2.297780 -1.731048  5.908837 -0.386345  0.616242       0  
2  5.360850  0.351993  2.940021  3.839160 -4.309402       0  
3  5.550100 -1.526796  0.138853  3.101430 -1.277378       0  
4 -3.229763  1.686909 -2.163896 -3.644622  6.510338       0  

[5 rows x 41 columns]
         V1        V2        V3        V4        V5        V6        V7  \
0 -0.613489 -3.819640  2.202302  1.300420 -1.184929 -4.495964 -1.835817   
1  0.389608 -0.512341  0.527053 -2.576776 -1.016766  2.235112 -0.441301   
2 -0.874861 -0.640632  4.084202 -1.590454  0.525855 -1.957592 -0.695367   
3  0.238384  1.458607  4.014528  2.534478  1.196987 -3.117330 -0.924035   
4  5.828225  2.768260 -1.234530  2.809264 -1.641648 -1.406698  0.568643   

         V8        V9       V10  ...       V32       V33       V34       V35  \
0  4.722989  1.206140 -0.341909  ...  2.291204 -5.411388  0.870073  0.574479   
1 -4.405744 -0.332869  1.966794  ... -2.474936  2.493582  0.315165  2.059288   
2  1.347309 -1.732348  0.466500  ... -1.318888 -2.997464  0.459664  0.619774   
3  0.269493  1.322436  0.702345  ...  3.517918 -3.074085 -0.284220  0.954576   
4  0.965043  1.918379 -2.774855  ...  1.773841 -1.501573 -2.226702  4.776830   

        V36       V37        V38       V39       V40  Target  
0  4.157191  1.428093 -10.511342  0.454664 -1.448363       0  
1  0.683859 -0.485452   5.128350  1.720744 -1.488235       0  
2  5.631504  1.323512  -1.752154  1.808302  1.675748       0  
3  3.029331 -1.367198  -3.412140  0.906000 -2.450889       0  
4 -6.559698 -0.805551  -0.276007 -3.858207 -0.537694       0  

[5 rows x 41 columns]
In [7]:
#View the shape of Training and Testing dataset

print(train_df.shape)
print(test_df.shape)

(20000, 41)
(5000, 41)
In [8]:
# View the datatypes of columns in Training and Testing dataset

print(train_df.info())
print(test_df.info())

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20000 entries, 0 to 19999
Data columns (total 41 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   V1      19982 non-null  float64
 1   V2      19982 non-null  float64
 2   V3      20000 non-null  float64
 3   V4      20000 non-null  float64
 4   V5      20000 non-null  float64
 5   V6      20000 non-null  float64
 6   V7      20000 non-null  float64
 7   V8      20000 non-null  float64
 8   V9      20000 non-null  float64
 9   V10     20000 non-null  float64
 10  V11     20000 non-null  float64
 11  V12     20000 non-null  float64
 12  V13     20000 non-null  float64
 13  V14     20000 non-null  float64
 14  V15     20000 non-null  float64
 15  V16     20000 non-null  float64
 16  V17     20000 non-null  float64
 17  V18     20000 non-null  float64
 18  V19     20000 non-null  float64
 19  V20     20000 non-null  float64
 20  V21     20000 non-null  float64
 21  V22     20000 non-null  float64
 22  V23     20000 non-null  float64
 23  V24     20000 non-null  float64
 24  V25     20000 non-null  float64
 25  V26     20000 non-null  float64
 26  V27     20000 non-null  float64
 27  V28     20000 non-null  float64
 28  V29     20000 non-null  float64
 29  V30     20000 non-null  float64
 30  V31     20000 non-null  float64
 31  V32     20000 non-null  float64
 32  V33     20000 non-null  float64
 33  V34     20000 non-null  float64
 34  V35     20000 non-null  float64
 35  V36     20000 non-null  float64
 36  V37     20000 non-null  float64
 37  V38     20000 non-null  float64
 38  V39     20000 non-null  float64
 39  V40     20000 non-null  float64
 40  Target  20000 non-null  int64  
dtypes: float64(40), int64(1)
memory usage: 6.3 MB
None
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 41 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   V1      4995 non-null   float64
 1   V2      4994 non-null   float64
 2   V3      5000 non-null   float64
 3   V4      5000 non-null   float64
 4   V5      5000 non-null   float64
 5   V6      5000 non-null   float64
 6   V7      5000 non-null   float64
 7   V8      5000 non-null   float64
 8   V9      5000 non-null   float64
 9   V10     5000 non-null   float64
 10  V11     5000 non-null   float64
 11  V12     5000 non-null   float64
 12  V13     5000 non-null   float64
 13  V14     5000 non-null   float64
 14  V15     5000 non-null   float64
 15  V16     5000 non-null   float64
 16  V17     5000 non-null   float64
 17  V18     5000 non-null   float64
 18  V19     5000 non-null   float64
 19  V20     5000 non-null   float64
 20  V21     5000 non-null   float64
 21  V22     5000 non-null   float64
 22  V23     5000 non-null   float64
 23  V24     5000 non-null   float64
 24  V25     5000 non-null   float64
 25  V26     5000 non-null   float64
 26  V27     5000 non-null   float64
 27  V28     5000 non-null   float64
 28  V29     5000 non-null   float64
 29  V30     5000 non-null   float64
 30  V31     5000 non-null   float64
 31  V32     5000 non-null   float64
 32  V33     5000 non-null   float64
 33  V34     5000 non-null   float64
 34  V35     5000 non-null   float64
 35  V36     5000 non-null   float64
 36  V37     5000 non-null   float64
 37  V38     5000 non-null   float64
 38  V39     5000 non-null   float64
 39  V40     5000 non-null   float64
 40  Target  5000 non-null   int64  
dtypes: float64(40), int64(1)
memory usage: 1.6 MB
None
In [9]:
#checking for duplicate values in both Training and Testing dataset

print(train_df.duplicated().sum())
print(test_df.duplicated().sum())

0
0
In [10]:
#Checking for missing values in both Training and Testing dataset

print(train_df.isnull().sum())
print(test_df.isnull().sum())

V1        18
V2        18
V3         0
V4         0
V5         0
V6         0
V7         0
V8         0
V9         0
V10        0
V11        0
V12        0
V13        0
V14        0
V15        0
V16        0
V17        0
V18        0
V19        0
V20        0
V21        0
V22        0
V23        0
V24        0
V25        0
V26        0
V27        0
V28        0
V29        0
V30        0
V31        0
V32        0
V33        0
V34        0
V35        0
V36        0
V37        0
V38        0
V39        0
V40        0
Target     0
dtype: int64
V1        5
V2        6
V3        0
V4        0
V5        0
V6        0
V7        0
V8        0
V9        0
V10       0
V11       0
V12       0
V13       0
V14       0
V15       0
V16       0
V17       0
V18       0
V19       0
V20       0
V21       0
V22       0
V23       0
V24       0
V25       0
V26       0
V27       0
V28       0
V29       0
V30       0
V31       0
V32       0
V33       0
V34       0
V35       0
V36       0
V37       0
V38       0
V39       0
V40       0
Target    0
dtype: int64
In [11]:
#View the rows with missing values in column V1

train_df.loc[train_df['V1'].isnull() == True]
Out[11]:
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ... V32 V33 V34 V35 V36 V37 V38 V39 V40 Target
89 NaN -3.961403 2.787804 -4.712526 -3.007329 -1.541245 -0.881148 1.476656 0.574700 -1.100884 ... -8.326069 -5.140552 1.121314 -0.305907 5.315007 3.750044 -5.631174 2.372485 2.195956 0
5941 NaN 1.008391 1.227702 5.397082 0.064230 -2.706919 -2.028368 0.534046 3.006797 -2.362238 ... 1.869502 -3.115298 -0.550197 1.713781 -2.256960 0.410992 -3.434400 -1.299388 -1.768734 0
6317 NaN -5.205346 1.997652 -3.707913 -1.042200 -1.593126 -2.653309 0.852280 -1.310489 2.406924 ... 3.074149 -0.067649 -0.277521 3.196840 7.016205 1.302334 -4.580096 2.956254 -2.363150 0
6464 NaN 2.146202 5.004415 4.192063 1.427887 -6.438263 -0.931339 3.794120 -0.683032 -0.738941 ... 5.231472 -5.113312 1.745687 2.587189 3.990777 0.610716 -4.273457 1.864568 -3.599079 0
7073 NaN 2.534010 2.762821 -1.673718 -1.942214 -0.029961 0.911323 -3.199743 2.948610 -0.413229 ... -4.887077 -2.611526 -1.500807 2.036186 -0.828979 -1.369591 0.572366 -0.132183 -0.322007 0
8431 NaN -1.398710 -2.008106 -1.750341 0.932279 -1.290327 -0.270476 4.458834 -2.776270 -1.211766 ... 4.560244 -0.420834 -2.037313 1.109793 1.520594 2.113872 -2.252571 -0.939249 2.542411 0
8439 NaN -3.840585 0.197220 4.147789 1.151400 -0.993298 -4.732363 0.558966 -0.926683 0.457914 ... 6.818725 3.451213 0.241818 3.215765 1.203210 1.274857 -1.921229 0.578890 -2.837521 0
11156 NaN -0.666978 3.715829 4.934000 1.667596 -4.356097 -2.823137 0.373175 -0.709951 2.177428 ... 6.663446 -2.897697 3.068461 2.486862 4.808548 0.069305 -1.215784 3.013674 -5.972586 0
11287 NaN -2.561519 -0.180836 -7.194814 -1.043832 1.384845 1.306093 1.559192 -2.992173 1.274543 ... -2.531655 0.560392 -1.153884 -0.019205 4.065248 0.978880 -0.571288 0.630374 3.919467 0
11456 NaN 1.299595 4.382858 1.583219 -0.076564 0.658770 -1.638530 -4.814763 -0.914819 2.811808 ... 1.772287 5.755242 1.203739 5.663939 0.413630 -2.643934 5.529745 2.104536 -4.945350 0
12221 NaN -2.326319 -0.051978 0.615063 -0.895755 -2.437003 0.349826 2.092611 -2.933523 2.291272 ... 0.134995 -5.183424 5.251667 0.716371 3.210930 1.641985 1.543559 1.805163 -2.039510 0
12447 NaN 0.752613 -0.271099 1.301204 2.038697 -1.485203 -0.411939 0.980629 0.810336 -0.065120 ... 4.410397 -2.208567 -1.358706 -1.725697 1.679060 -0.208564 -2.335547 0.112248 -0.542931 0
13086 NaN 2.056243 3.330642 2.741497 2.783166 -0.444191 -2.015376 -0.887154 -1.110920 0.025289 ... 5.112126 4.675408 -1.709632 2.429762 0.996644 -1.190509 1.207054 0.511023 -0.884200 0
13411 NaN 2.704511 4.587169 1.867930 2.050133 -0.925076 -1.669496 -1.653803 -0.243383 -0.317316 ... 2.527207 3.625279 -1.200200 2.328028 1.666937 -0.943228 0.946846 1.655145 -1.665439 0
14202 NaN 7.038653 2.144536 -3.201788 4.112972 3.375972 -1.337179 -4.546371 1.941427 -5.466593 ... 0.157778 9.768106 -10.258190 0.513864 -1.974958 -0.029436 3.127486 0.009482 4.538125 0
15520 NaN 1.382556 3.236896 -3.818363 -1.917264 0.437686 1.347540 -2.036067 1.155712 0.306502 ... -5.414599 -0.896510 -1.057864 1.417365 1.161990 -1.147123 -0.048258 0.604532 0.814557 0
16576 NaN 3.933815 -0.761930 2.651889 1.753614 -0.554092 1.829107 -0.105409 -3.737081 1.036776 ... 3.486408 1.028094 2.845747 1.744060 -1.999615 -0.783041 8.698449 0.352489 -2.005397 0
18104 NaN 1.492173 2.659206 0.222784 -0.303648 -1.347322 0.044309 -0.159095 1.108116 -0.572670 ... -1.007343 -2.229579 -0.870845 1.299595 0.667952 -0.503349 -1.485419 -0.153722 0.156501 0

18 rows × 41 columns

In [12]:
#To view rows with missing values in column V2

train_df.loc[train_df['V2'].isnull() == True]
Out[12]:
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ... V32 V33 V34 V35 V36 V37 V38 V39 V40 Target
613 -2.048681 NaN -1.623885 -3.324224 0.152256 0.600157 -1.812802 0.852194 -1.522600 0.211071 ... 3.264218 2.379064 -2.457084 1.719365 2.537010 1.701780 -1.434535 0.597365 0.739238 0
2236 -3.760658 NaN 0.194954 -1.637958 1.261479 -1.573947 -3.685700 1.575651 -0.309823 -0.137656 ... 7.620821 1.695061 -3.956354 2.707644 4.657387 1.619307 -5.537285 1.246650 -1.162793 0
2508 -1.430888 NaN 0.659576 -2.876402 1.150137 -0.785760 -1.560174 2.898635 -2.346989 -0.217607 ... 6.279266 3.323914 -4.047760 3.119220 3.336260 0.603524 -3.781725 -0.157478 1.503298 0
4653 5.465769 NaN 4.540947 -2.916550 0.399752 2.798925 0.029477 -7.334071 1.122874 1.695269 ... -1.535753 4.596212 -4.103525 4.295524 0.152672 -3.726700 6.562692 0.706452 -0.461696 0
6810 -2.631454 NaN 2.330188 1.090080 0.603973 -1.139383 -0.690121 -1.358935 0.355568 -1.189176 ... -0.950215 0.209717 0.448728 1.046063 0.536937 0.763187 1.728621 1.885821 -1.701774 0
7788 -4.203459 NaN 2.953868 0.584466 4.103940 -0.639211 -2.810799 -0.112492 -1.362768 -0.800101 ... 12.522374 9.502488 -7.152953 5.668769 1.249833 -2.158520 -0.954461 -0.002385 -1.546808 0
8483 -4.484232 NaN 1.200644 -2.042064 2.779443 -0.801748 -5.403548 -1.225314 1.485831 -0.974382 ... 9.467401 4.281421 -7.588117 3.266825 5.232311 1.278590 -5.370513 1.984130 -1.643391 0
8894 3.263555 NaN 8.446574 -3.253218 -3.417978 -2.995838 -0.669271 -0.161283 -0.666870 3.133527 ... -4.242730 -3.122680 2.522415 5.283805 7.291310 -0.867555 -4.315230 3.124488 -2.393239 0
8947 -3.793170 NaN 0.719610 2.306296 0.934728 -0.984321 0.504867 -0.441008 -2.767177 1.734671 ... 1.527720 -0.496910 3.789736 1.130689 0.618278 -0.111146 5.708912 1.542366 -2.481019 0
9362 2.662045 NaN 2.980068 4.430762 -0.237769 0.671919 0.380068 -7.646684 4.434754 -0.746393 ... -5.493590 -1.104656 1.224987 0.975596 -4.794411 -2.269039 7.670648 0.824983 -3.929104 0
9425 -2.354134 NaN 2.053893 0.811660 2.540366 -0.924875 -0.208380 -0.562864 -0.140210 -2.146916 ... -0.621103 -0.896509 -1.181480 -1.236617 1.237120 1.228277 2.073727 1.223874 1.472175 0
9848 -1.763501 NaN 2.845012 -2.753083 -0.811848 -0.101166 -1.382141 -1.105042 -0.054339 0.159742 ... -2.158880 1.859682 -0.337278 1.509300 3.408411 0.922594 -1.502959 2.514666 -0.793574 0
11637 -2.270541 NaN 1.710061 1.157522 -0.355177 -5.449480 -0.786321 3.936176 -1.576138 0.800881 ... 2.651480 -8.429332 3.511387 1.500102 5.552380 2.588580 -3.453418 2.324339 -2.760081 0
12339 -1.663687 NaN -0.712286 -4.346935 1.391670 -0.093951 -2.163175 -0.380573 0.031191 -0.658845 ... 0.306588 -2.690990 -3.111879 -1.596402 5.821108 3.462033 -1.736752 2.291092 2.240769 0
15913 0.768122 NaN 5.296110 0.043018 -1.173729 -2.248575 0.956395 -0.089941 -0.241678 -1.061413 ... -7.720265 -4.518617 3.182253 0.453452 2.175494 1.261707 0.892630 2.026732 0.632903 0
18342 -0.928572 NaN 2.375506 -1.236914 3.228744 -2.100088 -2.189908 0.588644 1.955973 -5.008491 ... 1.613181 -1.820569 -6.664808 -0.455080 3.054891 2.935276 -3.791135 0.863011 3.335753 0
18343 -2.377369 NaN -0.009173 -1.471979 1.295482 0.724894 -1.122797 -3.190475 3.250575 -4.861648 ... -5.093149 0.439355 -3.167241 -2.713266 -0.592845 3.229219 1.315635 2.282838 1.151589 0
18907 -0.119181 NaN 3.657612 -1.231802 1.946873 -0.119089 0.652414 -1.490208 -0.033631 -2.556604 ... -4.670353 -0.593916 -1.650592 -1.405071 1.531267 1.079147 2.832949 1.450781 3.232659 0

18 rows × 41 columns

In [13]:
#since the rows where values are missing are in random order and also the total number of rows where data is missing = 36 which is 0.18 % of total rows (20,000). hence drop the rows that have missing data in column V1 and V2

train_df = train_df.dropna(subset=['V1','V2'])
train_df.isnull().sum()
train_df.shape
Out[13]:
(19964, 41)
In [14]:
#since the rows where values are missing are in random order and also the total number of rows where data is missing = 11 which is 0.22 % of total rows (5,000).
#hence drop the rows that have missing data in column V1 and V2 in testing dataset

test_df = test_df.dropna(subset=['V1','V2'])
test_df.isnull().sum()
test_df.shape
Out[14]:
(4989, 41)
In [15]:
#To get a statistical summary of the dataset

train_df.describe().T
Out[15]:
count mean std min 25% 50% 75% max
V1 19964.0 -0.271156 3.442155 -11.876451 -2.737380 -0.746118 1.841050 15.493002
V2 19964.0 0.440507 3.150895 -12.319951 -1.640580 0.470788 2.544489 13.089269
V3 19964.0 2.485454 3.390736 -10.708139 0.207329 2.255786 4.569975 17.090919
V4 19964.0 -0.082647 3.432260 -15.082052 -2.345815 -0.135241 2.130757 13.236381
V5 19964.0 -0.055028 2.105122 -8.603361 -1.537210 -0.103365 1.339176 8.133797
V6 19964.0 -0.995463 2.041211 -10.227147 -2.347919 -1.000852 0.380444 6.975847
V7 19964.0 -0.879029 1.761840 -7.949681 -2.031207 -0.917179 0.223695 8.006091
V8 19964.0 -0.548384 3.296765 -15.657561 -2.647433 -0.390276 1.726303 11.679495
V9 19964.0 -0.016578 2.161033 -8.596313 -1.495193 -0.066895 1.409579 8.137580
V10 19964.0 -0.012416 2.193498 -9.853957 -1.412478 0.102264 1.477754 8.108472
V11 19964.0 -1.894280 3.124513 -14.832058 -3.922404 -1.920594 0.119336 11.826433
V12 19964.0 1.603980 2.930702 -12.948007 -0.396660 1.507160 3.571182 15.080698
V13 19964.0 1.580054 2.875043 -13.228247 -0.224047 1.636446 3.459854 15.419616
V14 19964.0 -0.950604 1.789884 -7.738593 -2.170457 -0.957426 0.270452 5.670664
V15 19964.0 -2.413627 3.355168 -16.416606 -4.413297 -2.382285 -0.358942 12.246455
V16 19964.0 -2.925148 4.223953 -20.374158 -5.638282 -2.683801 -0.092976 13.583212
V17 19964.0 -0.134968 3.345533 -14.091184 -2.215393 -0.015050 2.066289 16.756432
V18 19964.0 1.187689 2.592117 -11.643994 -0.405252 0.881894 2.569720 13.179863
V19 19964.0 1.183252 3.396921 -13.491784 -1.047863 1.281276 3.492482 13.237742
V20 19964.0 0.023871 3.669621 -13.922659 -2.433116 0.033415 2.513245 16.052339
V21 19964.0 -3.611479 3.569582 -17.956231 -5.932277 -3.532459 -1.265252 13.840473
V22 19964.0 0.951407 1.651725 -10.122095 -0.118339 0.974687 2.025396 7.409856
V23 19964.0 -0.367829 4.032946 -14.866128 -3.102770 -0.263199 2.450676 14.458734
V24 19964.0 1.133309 3.912658 -16.387147 -1.468354 0.965578 3.544004 17.163291
V25 19964.0 -0.001582 2.017157 -8.228266 -1.364949 0.026305 1.397890 8.223389
V26 19964.0 1.873105 3.434893 -11.834271 -0.337863 1.950531 4.128337 16.836410
V27 19964.0 -0.610981 4.369482 -14.904939 -3.650416 -0.878398 2.189664 17.560404
V28 19964.0 -0.882974 1.917511 -9.269489 -2.170332 -0.891073 0.375977 6.527643
V29 19964.0 -0.985055 2.684795 -12.579469 -2.787329 -1.177999 0.629773 10.722055
V30 19964.0 -0.015346 3.006321 -14.796047 -1.866374 0.184650 2.037034 12.505812
V31 19964.0 0.486920 3.462907 -13.722760 -1.818319 0.491126 2.731102 17.255090
V32 19964.0 0.302523 5.501439 -19.876502 -3.421328 0.049207 3.760799 23.633187
V33 19964.0 0.049735 3.574591 -16.898353 -2.242489 -0.064830 2.254949 16.692486
V34 19964.0 -0.461671 3.183388 -17.985094 -2.136799 -0.253433 1.437620 14.358213
V35 19964.0 2.230852 2.938409 -15.349803 0.336096 2.101516 4.068591 15.291065
V36 19964.0 1.513850 3.802518 -14.833178 -0.948662 1.566526 3.983750 19.329576
V37 19964.0 0.010491 1.788173 -5.478350 -1.257895 -0.130650 1.173907 7.467006
V38 19964.0 -0.344184 3.948539 -17.375002 -2.987116 -0.315217 2.280863 15.289923
V39 19964.0 0.890116 1.753959 -6.438880 -0.273597 0.919204 2.057540 7.759877
V40 19964.0 -0.876240 3.013024 -11.023935 -2.943705 -0.921977 1.119780 10.654265
Target 19964.0 0.055600 0.229154 0.000000 0.000000 0.000000 0.000000 1.000000
In [16]:
# Check for class distribution of the target variable by looking at the count of observations for each class in Training Dataset

class_distribution = train_df['Target'].value_counts(normalize=True) * 100
print(class_distribution)

Target
0    94.439992
1     5.560008
Name: proportion, dtype: float64
In [17]:
# Check for class distribution of the target variable by looking at the count of observations for each class in Testing Dataset

class_distribution = test_df['Target'].value_counts(normalize=True) * 100
print(class_distribution)

Target
0    94.347565
1     5.652435
Name: proportion, dtype: float64

Observations:

Dataset Size:

  1. The training set contains 20,000 observations, each with 41 columns (40 predictors and 1 target variable).
  2. The test set includes 5,000 observations, also with 41 columns (40 predictors and 1 target variable).

Data Types:

  1. Both datasets primarily consist of numerical features (float64), with the target variable being an integer (int64).

Missing Values:

  1. In the training set, two variables (V1 and V2) have 18 missing values each.
  2. In the test set, V1 has 5 missing values, and V2 has 6 missing values.
  3. No missing values are present in the target variable of both datasets.
  4. Since the rows where values are missing are in random order and also the total number of rows where data is missing = 36 which is 0.18 % of total rows (20,000). Hence dropped the rows that have missing data in column V1 and V2 in Traning dataset.

Data Integrity:

  1. There are no duplicate rows in either the training or the test dataset,indicating good data integrity.

Class Imbalance:

Both training and testing target variables have imbalanced distribution of the target variables classes where:

  1. About 94% of the observations belong to the negative class (0), which represents "Non-failure" cases.
  2. About 5% of the observations belong to the positive class (1), representing "Failure" cases.

EDA - Univariate Analysis¶

In [18]:
#define a function to plot the boxplot and histogram for all numerical columns

def histogram_boxplot(data, feature, figsize=(15,10), kde =True):
  fig,(axis_box,axis_hist) = plt.subplots(nrows=2,sharex=True,gridspec_kw={'height_ratios':(0.25,0.75)},figsize=figsize)
  #creating 2 subplots
  sns.boxplot(data=data,x=feature,ax=axis_box,showmeans=True,color='violet')
  sns.histplot(data=data,x=feature,kde=kde,ax=axis_hist,color='red')
  axis_hist.axvline(data[feature].mean(),color='green',linestyle='dotted')
  axis_hist.axvline(data[feature].median(),color='black',linestyle='dashed')
In [19]:
#Create the plots for all the independant columns

for col in train_df.columns:
  if col != 'Target':
    histogram_boxplot(train_df,col)
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [20]:
#Code to explain the skewness of the data in detail
from scipy.stats import skew

# Skewness
skewness = train_df.apply(skew)
for column in train_df.columns[:-1]:  # Exclude target column
    skew_val = skewness[column]
    if skew_val < -1:
        skew_type = "highly negatively skewed"
    elif -1 <= skew_val < -0.5:
        skew_type = "moderately negatively skewed"
    elif -0.5 <= skew_val < 0.5:
        skew_type = "approximately symmetric"
    elif 0.5 <= skew_val < 1:
        skew_type = "moderately positively skewed"
    else:
        skew_type = "highly positively skewed"

    print(f'The distribution of {column} is {skew_type} with a skewness of {skew_val:.2f}.')

The distribution of V1 is moderately positively skewed with a skewness of 0.54.
The distribution of V2 is approximately symmetric with a skewness of -0.04.
The distribution of V3 is approximately symmetric with a skewness of 0.32.
The distribution of V4 is approximately symmetric with a skewness of 0.10.
The distribution of V5 is approximately symmetric with a skewness of 0.13.
The distribution of V6 is approximately symmetric with a skewness of -0.03.
The distribution of V7 is approximately symmetric with a skewness of 0.18.
The distribution of V8 is approximately symmetric with a skewness of -0.26.
The distribution of V9 is approximately symmetric with a skewness of 0.10.
The distribution of V10 is approximately symmetric with a skewness of -0.25.
The distribution of V11 is approximately symmetric with a skewness of 0.04.
The distribution of V12 is approximately symmetric with a skewness of 0.09.
The distribution of V13 is approximately symmetric with a skewness of -0.18.
The distribution of V14 is approximately symmetric with a skewness of 0.00.
The distribution of V15 is approximately symmetric with a skewness of -0.06.
The distribution of V16 is approximately symmetric with a skewness of -0.21.
The distribution of V17 is approximately symmetric with a skewness of -0.14.
The distribution of V18 is approximately symmetric with a skewness of 0.30.
The distribution of V19 is approximately symmetric with a skewness of -0.14.
The distribution of V20 is approximately symmetric with a skewness of -0.03.
The distribution of V21 is approximately symmetric with a skewness of -0.01.
The distribution of V22 is approximately symmetric with a skewness of -0.17.
The distribution of V23 is approximately symmetric with a skewness of -0.11.
The distribution of V24 is approximately symmetric with a skewness of 0.22.
The distribution of V25 is approximately symmetric with a skewness of -0.09.
The distribution of V26 is approximately symmetric with a skewness of -0.09.
The distribution of V27 is approximately symmetric with a skewness of 0.31.
The distribution of V28 is approximately symmetric with a skewness of 0.04.
The distribution of V29 is approximately symmetric with a skewness of 0.34.
The distribution of V30 is approximately symmetric with a skewness of -0.36.
The distribution of V31 is approximately symmetric with a skewness of 0.01.
The distribution of V32 is approximately symmetric with a skewness of 0.24.
The distribution of V33 is approximately symmetric with a skewness of 0.14.
The distribution of V34 is approximately symmetric with a skewness of -0.36.
The distribution of V35 is approximately symmetric with a skewness of 0.08.
The distribution of V36 is approximately symmetric with a skewness of -0.03.
The distribution of V37 is approximately symmetric with a skewness of 0.32.
The distribution of V38 is approximately symmetric with a skewness of -0.02.
The distribution of V39 is approximately symmetric with a skewness of -0.05.
The distribution of V40 is approximately symmetric with a skewness of 0.10.

Observations:

Independent variables:

The histograms indicate that the data for these variables is approximately normally distributed with mean and median fairly close for most, further suggesting a normal distribution. The boxplots show dots outside of the whiskers, which represent outliers. These are observations that fall significantly higher or lower than the majority of the data.

EDA - Bivariate Analysis¶

In [21]:
# Heatmap to visualise the correlation of numerical variables

plt.figure(figsize=(25,25))
# Generate a mask for the upper triangle
mask = np.triu(np.ones_like(train_df.corr(), dtype=bool))
sns.heatmap(train_df.corr(),mask=mask,annot=True,cmap='coolwarm',vmax=1,vmin=-1,fmt=".2f")
Out[21]:
<Axes: >
No description has been provided for this image
In [22]:
#Create the scatter plots for all the 40 columns

target_variable = 'Target'
for column in train_df.columns:
    if column != target_variable:
        plt.figure()
        plt.scatter(train_df[column], train_df[target_variable], c=train_df[target_variable], cmap='bwr', marker='o')
        plt.xlabel(column)
        plt.ylabel(target_variable)
        plt.title(f'Scatter Plot of {column} vs. {target_variable}')
        plt.show()
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [23]:
#Box plot to see the distribution of columns w.r.t classes in target column

fig, axes = plt.subplots(nrows = 8, ncols = 5, figsize = (14, 24))
axes = axes.flat

for i,col in enumerate(train_df.columns[:-1]):
    sns.boxplot(train_df, x = 'Target', y = col, ax = axes[i])

    axes[i].set_title(f'{col} by Target', fontsize = 9, fontweight = 'bold', color = 'black')
    axes[i].set_xlabel('')

fig.suptitle('Distribution of Numerical Variables by Target', fontsize = 12, fontweight = 'bold', color = 'darkblue', y = 0.93)
fig.tight_layout()
fig.subplots_adjust(top = 0.9)
fig.show()
No description has been provided for this image

Observations:

Lots of Highly Correlated Features are there both positive and negative. These may represent redundancy within the dataset which means that one feature can be predicted from the other with a high degree of accuracy. In other words, these features carry similar information or signals about the data.

  1. Some of the strong correlations are seen between - (V11,V6), (V12,V8), (V15,V7), (V16,V8), (V36,V39), (V19,V34), (V24,V32), (V11,V29), (V26,V2)

  2. Columns with strong correlation w.r.t Target variable are - V7, V11, V15, V16, V21, V28, V34.

Data preprocessing¶

In [24]:
#checking the split of target column

train_df.shape
train_df['Target'].value_counts()
#test_df['Target'].value_counts()
Out[24]:
count
Target
0 18854
1 1110

In [25]:
#separate the Dependant and Independant variables in Train and Test Dataframe

# Separate features and target for training data
X_train_temp = train_df.drop(columns=['Target'])
y_train_temp = train_df['Target']

# Separate features and target for testing data
X_test = test_df.drop(columns=['Target'])
y_test = test_df['Target']
In [26]:
#Split Training Data into Training and Validation Sets

X_train, X_val, y_train, y_val = train_test_split(X_train_temp, y_train_temp, test_size=0.5, random_state=42)
In [27]:
print("Number of rows in train data =", X_train.shape[0])
print("Number of rows in validation data =", X_val.shape[0])
print("Number of rows in test data =", X_test.shape[0])

Number of rows in train data = 9982
Number of rows in validation data = 9982
Number of rows in test data = 4989
In [28]:
X_train.shape
Out[28]:
(9982, 40)
In [29]:
#Normalising the data using standard scalar
#Transform numerical columns in x_train, x_val, and x_test using the fitted scaler

scaler = StandardScaler()
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_val = scaler.transform(X_val)
X_test = scaler.transform(X_test)
In [30]:
#compute class weights
from sklearn.utils.class_weight import compute_class_weight

class_weights = compute_class_weight(class_weight='balanced',classes=np.unique(y_train),y=y_train)
class_weights = dict(enumerate(class_weights))

Utility Functions¶

In [31]:
#Define a function to compute the different metrics to check the performance of the Neural Network model

def model_performance(model, predictors, target, threshold=0.5):

  predictions = model.predict(predictors) > threshold

  accuracy = accuracy_score(target, predictions)
  recall= recall_score(target, predictions, average = 'weighted')
  precision = precision_score(target, predictions, average = 'weighted')
  f1 = f1_score(target, predictions, average = 'weighted')

  #creating a dataframe of metrics
  perf_metrics_df = pd.DataFrame({'Accuracy': accuracy, 'Recall': recall, 'Precision': precision, 'F1 Score': f1}, index=[0])

  return perf_metrics_df

columns = ["hidden_layers","neurons_HL","activation_ftn_HL","epochs","batch_size","Batch_Norm","Weight_Ini","dropout","optimizer",
           "time(secs)","Train_loss","Validation_loss",
           "Train_Precision","Validation_Precision","Train_Recall","Validation_Recall",
           "Test_Precision","Test_Recall"]
results = pd.DataFrame(columns=columns)
In [32]:
#Define a fucntion to plot the metric chosen for training and validation data

def plot_training_history(history, metric):

  if metric not in history.history or f"val_{metric}" not in history.history:
    print(f"The metric {metric} is not available in the history.")
    return

  plt.figure(figsize=(12, 6))
  plt.plot(history.history[metric], label=f"Training {metric}")
  plt.plot(history.history[f'val_{metric}'], label=f"Validation {metric}")
  plt.title(f"Training and Validation {metric} over Epochs")
  plt.xlabel('Epochs')
  plt.ylabel(metric.capitalize())
  plt.legend(loc='best')
  plt.grid(True)
  plt.show()

Model Building¶

In [33]:
#Defining the list of metrics to be used for all the models

metrics = [tf.keras.metrics.Precision(name="precision"), tf.keras.metrics.Recall(name="recall")]
Model 0 - Shallow Neural Network to check the linearity of the model¶
  1. No Hidden layers
  2. SGD
  3. Loss function - Binary Cross entropy
  4. Epochs - 10
  5. Batch size - Input rows
In [34]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model0 = Sequential()
model0.add(Dense(1,input_dim = X_train.shape[1]))
model0.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 1)                 41        
                                                                 
=================================================================
Total params: 41 (164.00 Byte)
Trainable params: 41 (164.00 Byte)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [35]:
#Add the necessary configurations and fit the model

optimizer = keras.optimizers.SGD()
model0.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 10
batch_size = X_train.shape[0]
start = time.time()
history = model0.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val))
end = time.time()

Epoch 1/10
1/1 [==============================] - 1s 724ms/step - loss: 4.6068 - precision: 0.0319 - recall: 0.2228 - val_loss: 4.5683 - val_precision: 0.0339 - val_recall: 0.2170
Epoch 2/10
1/1 [==============================] - 0s 49ms/step - loss: 4.4507 - precision: 0.0298 - recall: 0.2041 - val_loss: 4.3926 - val_precision: 0.0347 - val_recall: 0.2170
Epoch 3/10
1/1 [==============================] - 0s 39ms/step - loss: 4.2698 - precision: 0.0310 - recall: 0.2060 - val_loss: 4.2258 - val_precision: 0.0357 - val_recall: 0.2170
Epoch 4/10
1/1 [==============================] - 0s 34ms/step - loss: 4.1031 - precision: 0.0314 - recall: 0.2041 - val_loss: 4.0429 - val_precision: 0.0367 - val_recall: 0.2170
Epoch 5/10
1/1 [==============================] - 0s 49ms/step - loss: 3.9199 - precision: 0.0320 - recall: 0.2022 - val_loss: 3.7776 - val_precision: 0.0391 - val_recall: 0.2222
Epoch 6/10
1/1 [==============================] - 0s 48ms/step - loss: 3.6104 - precision: 0.0331 - recall: 0.2004 - val_loss: 3.0733 - val_precision: 0.0530 - val_recall: 0.2691
Epoch 7/10
1/1 [==============================] - 0s 45ms/step - loss: 2.9496 - precision: 0.0482 - recall: 0.2584 - val_loss: 2.7615 - val_precision: 0.0550 - val_recall: 0.2587
Epoch 8/10
1/1 [==============================] - 0s 32ms/step - loss: 2.6542 - precision: 0.0517 - recall: 0.2566 - val_loss: 2.3993 - val_precision: 0.0566 - val_recall: 0.2448
Epoch 9/10
1/1 [==============================] - 0s 33ms/step - loss: 2.3250 - precision: 0.0517 - recall: 0.2303 - val_loss: 2.0632 - val_precision: 0.0578 - val_recall: 0.2292
Epoch 10/10
1/1 [==============================] - 0s 47ms/step - loss: 2.0315 - precision: 0.0543 - recall: 0.2191 - val_loss: 1.6960 - val_precision: 0.0569 - val_recall: 0.1892
In [36]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [37]:
output = model_performance(model0, X_test, y_test)
output

156/156 [==============================] - 0s 1ms/step
Out[37]:
Accuracy Recall Precision F1 Score
0 0.764682 0.764682 0.890782 0.820497
In [38]:
results.loc[0] = ['-','-','-',epochs,batch_size,'-','-','-',"SGD",end-start,history.history['loss'][-1],history.history['val_loss'][-1],history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                  ,output['Precision'][0],output['Recall'][0]]
results
Out[38]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.05686 0.219101 0.189236 0.890782 0.764682
Model 1 - Shallow Network¶
  1. No Hidden layers
  2. SGD with learning rate - 0.01
  3. Loss function - Binary Cross entropy
  4. Epochs - 25
  5. Batch size - Input rows
In [39]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model1 = Sequential()
model1.add(Dense(1,input_dim = X_train.shape[1]))
model1.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 1)                 41        
                                                                 
=================================================================
Total params: 41 (164.00 Byte)
Trainable params: 41 (164.00 Byte)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [40]:
#Add the necessary configurations and fit the model

optimizer = keras.optimizers.SGD(learning_rate=0.01)
model1.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = X_train.shape[0]
start = time.time()
history = model1.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val))
end = time.time()

Epoch 1/25
1/1 [==============================] - 1s 651ms/step - loss: 4.3678 - precision: 0.0436 - recall: 0.2054 - val_loss: 4.1789 - val_precision: 0.0357 - val_recall: 0.1979
Epoch 2/25
1/1 [==============================] - 0s 45ms/step - loss: 4.0851 - precision: 0.0299 - recall: 0.1760 - val_loss: 3.9125 - val_precision: 0.0370 - val_recall: 0.1962
Epoch 3/25
1/1 [==============================] - 0s 32ms/step - loss: 3.7993 - precision: 0.0306 - recall: 0.1723 - val_loss: 3.6912 - val_precision: 0.0390 - val_recall: 0.1979
Epoch 4/25
1/1 [==============================] - 0s 48ms/step - loss: 3.6145 - precision: 0.0339 - recall: 0.1835 - val_loss: 3.4839 - val_precision: 0.0387 - val_recall: 0.1892
Epoch 5/25
1/1 [==============================] - 0s 48ms/step - loss: 3.4107 - precision: 0.0352 - recall: 0.1835 - val_loss: 3.2554 - val_precision: 0.0397 - val_recall: 0.1858
Epoch 6/25
1/1 [==============================] - 0s 48ms/step - loss: 3.1711 - precision: 0.0357 - recall: 0.1760 - val_loss: 3.0874 - val_precision: 0.0391 - val_recall: 0.1753
Epoch 7/25
1/1 [==============================] - 0s 54ms/step - loss: 3.0135 - precision: 0.0370 - recall: 0.1742 - val_loss: 2.9178 - val_precision: 0.0387 - val_recall: 0.1632
Epoch 8/25
1/1 [==============================] - 0s 48ms/step - loss: 2.8593 - precision: 0.0364 - recall: 0.1629 - val_loss: 2.4811 - val_precision: 0.0409 - val_recall: 0.1545
Epoch 9/25
1/1 [==============================] - 0s 49ms/step - loss: 2.4252 - precision: 0.0403 - recall: 0.1610 - val_loss: 2.0866 - val_precision: 0.0400 - val_recall: 0.1337
Epoch 10/25
1/1 [==============================] - 0s 50ms/step - loss: 2.0360 - precision: 0.0418 - recall: 0.1479 - val_loss: 1.8209 - val_precision: 0.0668 - val_recall: 0.2274
Epoch 11/25
1/1 [==============================] - 0s 51ms/step - loss: 1.7439 - precision: 0.0670 - recall: 0.2472 - val_loss: 1.2730 - val_precision: 0.0780 - val_recall: 0.2014
Epoch 12/25
1/1 [==============================] - 0s 36ms/step - loss: 1.2054 - precision: 0.0833 - recall: 0.2285 - val_loss: 0.9071 - val_precision: 0.0980 - val_recall: 0.1736
Epoch 13/25
1/1 [==============================] - 0s 49ms/step - loss: 0.8691 - precision: 0.1071 - recall: 0.2116 - val_loss: 0.7955 - val_precision: 0.1108 - val_recall: 0.1597
Epoch 14/25
1/1 [==============================] - 0s 48ms/step - loss: 0.7535 - precision: 0.1203 - recall: 0.1873 - val_loss: 0.7109 - val_precision: 0.1326 - val_recall: 0.1250
Epoch 15/25
1/1 [==============================] - 0s 47ms/step - loss: 0.6392 - precision: 0.1635 - recall: 0.1592 - val_loss: 0.6625 - val_precision: 0.1791 - val_recall: 0.1458
Epoch 16/25
1/1 [==============================] - 0s 48ms/step - loss: 0.5993 - precision: 0.1892 - recall: 0.1648 - val_loss: 0.6124 - val_precision: 0.2131 - val_recall: 0.1302
Epoch 17/25
1/1 [==============================] - 0s 51ms/step - loss: 0.5657 - precision: 0.2622 - recall: 0.1610 - val_loss: 0.5874 - val_precision: 0.2618 - val_recall: 0.1250
Epoch 18/25
1/1 [==============================] - 0s 56ms/step - loss: 0.5528 - precision: 0.3142 - recall: 0.1536 - val_loss: 0.5705 - val_precision: 0.2992 - val_recall: 0.1267
Epoch 19/25
1/1 [==============================] - 0s 65ms/step - loss: 0.5382 - precision: 0.3591 - recall: 0.1479 - val_loss: 0.5544 - val_precision: 0.3217 - val_recall: 0.1285
Epoch 20/25
1/1 [==============================] - 0s 62ms/step - loss: 0.5232 - precision: 0.3738 - recall: 0.1498 - val_loss: 0.5300 - val_precision: 0.3410 - val_recall: 0.1545
Epoch 21/25
1/1 [==============================] - 0s 70ms/step - loss: 0.5007 - precision: 0.3702 - recall: 0.1816 - val_loss: 0.5180 - val_precision: 0.3771 - val_recall: 0.1545
Epoch 22/25
1/1 [==============================] - 0s 62ms/step - loss: 0.4903 - precision: 0.3983 - recall: 0.1760 - val_loss: 0.5087 - val_precision: 0.4055 - val_recall: 0.1528
Epoch 23/25
1/1 [==============================] - 0s 44ms/step - loss: 0.4820 - precision: 0.4234 - recall: 0.1760 - val_loss: 0.4829 - val_precision: 0.4123 - val_recall: 0.1632
Epoch 24/25
1/1 [==============================] - 0s 44ms/step - loss: 0.4549 - precision: 0.4534 - recall: 0.2097 - val_loss: 0.4479 - val_precision: 0.3839 - val_recall: 0.2726
Epoch 25/25
1/1 [==============================] - 0s 43ms/step - loss: 0.4251 - precision: 0.4028 - recall: 0.3184 - val_loss: 0.4306 - val_precision: 0.4455 - val_recall: 0.2483
In [41]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [42]:
output = model_performance(model1, X_test, y_test)
output

156/156 [==============================] - 0s 2ms/step
Out[42]:
Accuracy Recall Precision F1 Score
0 0.940469 0.940469 0.928051 0.932551
In [43]:
results.loc[1] = ['-','-','-',epochs,batch_size,'-','-','-',"SGD_LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                  ,output['Precision'][0],output['Recall'][0]]
results
Out[43]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
Model 2 - Shallow Network¶
  1. No Hidden layers
  2. SGD with learning rate - 0.01
  3. Loss function - Binary Cross entropy
  4. Epochs - 50
  5. Batch size - Input rows
In [44]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model2 = Sequential()
model2.add(Dense(1,input_dim = X_train.shape[1]))
model2.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 1)                 41        
                                                                 
=================================================================
Total params: 41 (164.00 Byte)
Trainable params: 41 (164.00 Byte)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [45]:
#Add the necessary configurations and fit the model

optimizer = keras.optimizers.SGD(learning_rate=0.01)
model2.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 50
batch_size = X_train.shape[0]
start = time.time()
history = model2.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val))
end = time.time()

Epoch 1/50
1/1 [==============================] - 1s 662ms/step - loss: 3.3087 - precision: 0.1083 - recall: 0.3550 - val_loss: 3.0946 - val_precision: 0.0786 - val_recall: 0.4306
Epoch 2/50
1/1 [==============================] - 0s 34ms/step - loss: 3.0992 - precision: 0.0772 - recall: 0.4588 - val_loss: 2.8564 - val_precision: 0.0818 - val_recall: 0.4253
Epoch 3/50
1/1 [==============================] - 0s 34ms/step - loss: 2.8604 - precision: 0.0783 - recall: 0.4457 - val_loss: 2.5182 - val_precision: 0.0859 - val_recall: 0.4045
Epoch 4/50
1/1 [==============================] - 0s 48ms/step - loss: 2.4603 - precision: 0.0867 - recall: 0.4513 - val_loss: 2.0985 - val_precision: 0.0795 - val_recall: 0.3281
Epoch 5/50
1/1 [==============================] - 0s 51ms/step - loss: 2.0346 - precision: 0.0785 - recall: 0.3577 - val_loss: 1.7005 - val_precision: 0.0802 - val_recall: 0.2847
Epoch 6/50
1/1 [==============================] - 0s 40ms/step - loss: 1.6505 - precision: 0.0802 - recall: 0.3165 - val_loss: 1.3120 - val_precision: 0.0825 - val_recall: 0.2413
Epoch 7/50
1/1 [==============================] - 0s 32ms/step - loss: 1.2481 - precision: 0.0895 - recall: 0.2809 - val_loss: 1.0091 - val_precision: 0.0954 - val_recall: 0.2083
Epoch 8/50
1/1 [==============================] - 0s 48ms/step - loss: 0.9295 - precision: 0.1018 - recall: 0.2378 - val_loss: 0.8807 - val_precision: 0.1058 - val_recall: 0.1840
Epoch 9/50
1/1 [==============================] - 0s 48ms/step - loss: 0.8038 - precision: 0.1121 - recall: 0.2116 - val_loss: 0.7803 - val_precision: 0.1277 - val_recall: 0.1545
Epoch 10/50
1/1 [==============================] - 0s 47ms/step - loss: 0.6883 - precision: 0.1336 - recall: 0.1667 - val_loss: 0.7438 - val_precision: 0.1926 - val_recall: 0.1181
Epoch 11/50
1/1 [==============================] - 0s 31ms/step - loss: 0.6692 - precision: 0.2087 - recall: 0.1348 - val_loss: 0.7204 - val_precision: 0.2152 - val_recall: 0.1233
Epoch 12/50
1/1 [==============================] - 0s 47ms/step - loss: 0.6534 - precision: 0.2355 - recall: 0.1442 - val_loss: 0.7030 - val_precision: 0.2403 - val_recall: 0.1285
Epoch 13/50
1/1 [==============================] - 0s 33ms/step - loss: 0.6393 - precision: 0.2576 - recall: 0.1423 - val_loss: 0.6856 - val_precision: 0.2446 - val_recall: 0.1389
Epoch 14/50
1/1 [==============================] - 0s 53ms/step - loss: 0.6242 - precision: 0.2603 - recall: 0.1536 - val_loss: 0.6807 - val_precision: 0.2572 - val_recall: 0.1389
Epoch 15/50
1/1 [==============================] - 0s 49ms/step - loss: 0.6168 - precision: 0.2785 - recall: 0.1554 - val_loss: 0.6744 - val_precision: 0.2746 - val_recall: 0.1406
Epoch 16/50
1/1 [==============================] - 0s 35ms/step - loss: 0.6126 - precision: 0.2927 - recall: 0.1573 - val_loss: 0.6666 - val_precision: 0.2800 - val_recall: 0.1458
Epoch 17/50
1/1 [==============================] - 0s 49ms/step - loss: 0.6061 - precision: 0.2969 - recall: 0.1629 - val_loss: 0.6620 - val_precision: 0.2911 - val_recall: 0.1476
Epoch 18/50
1/1 [==============================] - 0s 47ms/step - loss: 0.6017 - precision: 0.3063 - recall: 0.1629 - val_loss: 0.6626 - val_precision: 0.3000 - val_recall: 0.1458
Epoch 19/50
1/1 [==============================] - 0s 33ms/step - loss: 0.5971 - precision: 0.3223 - recall: 0.1648 - val_loss: 0.6568 - val_precision: 0.3060 - val_recall: 0.1493
Epoch 20/50
1/1 [==============================] - 0s 46ms/step - loss: 0.5931 - precision: 0.3248 - recall: 0.1667 - val_loss: 0.6522 - val_precision: 0.3246 - val_recall: 0.1510
Epoch 21/50
1/1 [==============================] - 0s 34ms/step - loss: 0.5948 - precision: 0.3448 - recall: 0.1685 - val_loss: 0.6510 - val_precision: 0.3400 - val_recall: 0.1476
Epoch 22/50
1/1 [==============================] - 0s 34ms/step - loss: 0.5919 - precision: 0.3644 - recall: 0.1685 - val_loss: 0.6479 - val_precision: 0.3517 - val_recall: 0.1441
Epoch 23/50
1/1 [==============================] - 0s 32ms/step - loss: 0.5896 - precision: 0.3836 - recall: 0.1667 - val_loss: 0.6463 - val_precision: 0.3689 - val_recall: 0.1441
Epoch 24/50
1/1 [==============================] - 0s 34ms/step - loss: 0.5867 - precision: 0.4018 - recall: 0.1648 - val_loss: 0.6370 - val_precision: 0.3636 - val_recall: 0.1528
Epoch 25/50
1/1 [==============================] - 0s 31ms/step - loss: 0.5816 - precision: 0.4064 - recall: 0.1667 - val_loss: 0.5825 - val_precision: 0.3199 - val_recall: 0.2622
Epoch 26/50
1/1 [==============================] - 0s 31ms/step - loss: 0.5355 - precision: 0.3150 - recall: 0.2678 - val_loss: 0.5710 - val_precision: 0.3364 - val_recall: 0.2535
Epoch 27/50
1/1 [==============================] - 0s 47ms/step - loss: 0.5302 - precision: 0.3293 - recall: 0.2547 - val_loss: 0.5703 - val_precision: 0.3681 - val_recall: 0.2448
Epoch 28/50
1/1 [==============================] - 0s 39ms/step - loss: 0.5238 - precision: 0.3681 - recall: 0.2378 - val_loss: 0.5832 - val_precision: 0.4569 - val_recall: 0.2118
Epoch 29/50
1/1 [==============================] - 0s 56ms/step - loss: 0.5289 - precision: 0.4496 - recall: 0.2004 - val_loss: 0.5735 - val_precision: 0.4561 - val_recall: 0.2257
Epoch 30/50
1/1 [==============================] - 0s 44ms/step - loss: 0.5229 - precision: 0.4618 - recall: 0.2154 - val_loss: 0.5677 - val_precision: 0.4760 - val_recall: 0.2240
Epoch 31/50
1/1 [==============================] - 0s 42ms/step - loss: 0.5200 - precision: 0.4733 - recall: 0.2154 - val_loss: 0.5676 - val_precision: 0.4727 - val_recall: 0.2257
Epoch 32/50
1/1 [==============================] - 0s 32ms/step - loss: 0.5215 - precision: 0.4528 - recall: 0.2154 - val_loss: 0.5639 - val_precision: 0.4851 - val_recall: 0.2257
Epoch 33/50
1/1 [==============================] - 0s 35ms/step - loss: 0.5174 - precision: 0.4634 - recall: 0.2135 - val_loss: 0.5609 - val_precision: 0.4925 - val_recall: 0.2274
Epoch 34/50
1/1 [==============================] - 0s 38ms/step - loss: 0.5133 - precision: 0.4835 - recall: 0.2191 - val_loss: 0.5589 - val_precision: 0.5058 - val_recall: 0.2274
Epoch 35/50
1/1 [==============================] - 0s 34ms/step - loss: 0.5102 - precision: 0.4874 - recall: 0.2172 - val_loss: 0.5558 - val_precision: 0.5219 - val_recall: 0.2274
Epoch 36/50
1/1 [==============================] - 0s 50ms/step - loss: 0.5056 - precision: 0.5043 - recall: 0.2172 - val_loss: 0.5285 - val_precision: 0.4379 - val_recall: 0.2691
Epoch 37/50
1/1 [==============================] - 0s 55ms/step - loss: 0.4790 - precision: 0.4315 - recall: 0.2715 - val_loss: 0.5305 - val_precision: 0.4840 - val_recall: 0.2622
Epoch 38/50
1/1 [==============================] - 0s 52ms/step - loss: 0.4782 - precision: 0.4628 - recall: 0.2566 - val_loss: 0.5272 - val_precision: 0.5119 - val_recall: 0.2622
Epoch 39/50
1/1 [==============================] - 0s 51ms/step - loss: 0.4726 - precision: 0.4759 - recall: 0.2584 - val_loss: 0.5227 - val_precision: 0.4935 - val_recall: 0.2622
Epoch 40/50
1/1 [==============================] - 0s 50ms/step - loss: 0.4676 - precision: 0.4667 - recall: 0.2622 - val_loss: 0.5207 - val_precision: 0.5154 - val_recall: 0.2622
Epoch 41/50
1/1 [==============================] - 0s 48ms/step - loss: 0.4629 - precision: 0.4879 - recall: 0.2640 - val_loss: 0.5075 - val_precision: 0.4728 - val_recall: 0.2865
Epoch 42/50
1/1 [==============================] - 0s 48ms/step - loss: 0.4481 - precision: 0.4890 - recall: 0.2921 - val_loss: 0.5074 - val_precision: 0.5192 - val_recall: 0.2812
Epoch 43/50
1/1 [==============================] - 0s 50ms/step - loss: 0.4473 - precision: 0.5033 - recall: 0.2828 - val_loss: 0.5032 - val_precision: 0.5281 - val_recall: 0.2778
Epoch 44/50
1/1 [==============================] - 0s 48ms/step - loss: 0.4429 - precision: 0.5190 - recall: 0.2809 - val_loss: 0.5005 - val_precision: 0.5408 - val_recall: 0.2760
Epoch 45/50
1/1 [==============================] - 0s 31ms/step - loss: 0.4424 - precision: 0.5267 - recall: 0.2772 - val_loss: 0.4973 - val_precision: 0.5077 - val_recall: 0.2847
Epoch 46/50
1/1 [==============================] - 0s 33ms/step - loss: 0.4381 - precision: 0.5201 - recall: 0.2903 - val_loss: 0.4944 - val_precision: 0.5482 - val_recall: 0.2865
Epoch 47/50
1/1 [==============================] - 0s 34ms/step - loss: 0.4339 - precision: 0.5401 - recall: 0.2903 - val_loss: 0.4948 - val_precision: 0.5186 - val_recall: 0.2899
Epoch 48/50
1/1 [==============================] - 0s 50ms/step - loss: 0.4403 - precision: 0.5356 - recall: 0.2959 - val_loss: 0.4946 - val_precision: 0.5544 - val_recall: 0.2743
Epoch 49/50
1/1 [==============================] - 0s 42ms/step - loss: 0.4365 - precision: 0.5568 - recall: 0.2846 - val_loss: 0.4945 - val_precision: 0.5843 - val_recall: 0.2708
Epoch 50/50
1/1 [==============================] - 0s 40ms/step - loss: 0.4345 - precision: 0.5660 - recall: 0.2809 - val_loss: 0.4932 - val_precision: 0.5985 - val_recall: 0.2691
In [46]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [47]:
output = model_performance(model2, X_test, y_test)
output

156/156 [==============================] - 0s 1ms/step
Out[47]:
Accuracy Recall Precision F1 Score
0 0.951694 0.951694 0.942613 0.943551
In [48]:
results.loc[2] = ['-','-','-',epochs,batch_size,'-','-','-',"SGD-LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                  ,output['Precision'][0],output['Recall'][0]]

results
Out[48]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
Model 3 - Shallow Network¶
  1. No Hidden layers
  2. SGD with learning rate - 0.01
  3. Loss function - Binary Cross entropy
  4. Epochs - 25
  5. Batch size - 32
In [49]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model3 = Sequential()
model3.add(Dense(1,input_dim = X_train.shape[1]))
model3.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 1)                 41        
                                                                 
=================================================================
Total params: 41 (164.00 Byte)
Trainable params: 41 (164.00 Byte)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [50]:
#Add the necessary configurations and fit the model

optimizer = keras.optimizers.SGD(learning_rate=0.01)
model3.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model3.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val))
end = time.time()

Epoch 1/25
312/312 [==============================] - 2s 4ms/step - loss: 3.5195 - precision: 0.1445 - recall: 0.3946 - val_loss: 1.1456 - val_precision: 0.3056 - val_recall: 0.5747
Epoch 2/25
312/312 [==============================] - 1s 4ms/step - loss: 0.7386 - precision: 0.3907 - recall: 0.3446 - val_loss: 0.5384 - val_precision: 0.7912 - val_recall: 0.3420
Epoch 3/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4587 - precision: 0.6930 - recall: 0.4438 - val_loss: 0.6193 - val_precision: 0.5840 - val_recall: 0.2413
Epoch 4/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4201 - precision: 0.7864 - recall: 0.4345 - val_loss: 0.5043 - val_precision: 0.8421 - val_recall: 0.3611
Epoch 5/25
312/312 [==============================] - 1s 4ms/step - loss: 3.8959 - precision: 0.0889 - recall: 0.4213 - val_loss: 5.9308 - val_precision: 0.0569 - val_recall: 0.3767
Epoch 6/25
312/312 [==============================] - 1s 4ms/step - loss: 5.6630 - precision: 0.0700 - recall: 0.4925 - val_loss: 5.5041 - val_precision: 0.0736 - val_recall: 0.4635
Epoch 7/25
312/312 [==============================] - 1s 4ms/step - loss: 5.4921 - precision: 0.0717 - recall: 0.4906 - val_loss: 5.4136 - val_precision: 0.0697 - val_recall: 0.4288
Epoch 8/25
312/312 [==============================] - 1s 3ms/step - loss: 5.4270 - precision: 0.0728 - recall: 0.4906 - val_loss: 5.2505 - val_precision: 0.0759 - val_recall: 0.4549
Epoch 9/25
312/312 [==============================] - 1s 3ms/step - loss: 5.2429 - precision: 0.0768 - recall: 0.5056 - val_loss: 5.1243 - val_precision: 0.0753 - val_recall: 0.4392
Epoch 10/25
312/312 [==============================] - 1s 4ms/step - loss: 5.0941 - precision: 0.0763 - recall: 0.4831 - val_loss: 5.0282 - val_precision: 0.0727 - val_recall: 0.4097
Epoch 11/25
312/312 [==============================] - 1s 4ms/step - loss: 4.9839 - precision: 0.0755 - recall: 0.4663 - val_loss: 4.9481 - val_precision: 0.0724 - val_recall: 0.4010
Epoch 12/25
312/312 [==============================] - 1s 3ms/step - loss: 4.8353 - precision: 0.0772 - recall: 0.4625 - val_loss: 4.7934 - val_precision: 0.0755 - val_recall: 0.4062
Epoch 13/25
312/312 [==============================] - 1s 4ms/step - loss: 4.7986 - precision: 0.0634 - recall: 0.3633 - val_loss: 4.4633 - val_precision: 0.0561 - val_recall: 0.2656
Epoch 14/25
312/312 [==============================] - 1s 4ms/step - loss: 4.4105 - precision: 0.0553 - recall: 0.2828 - val_loss: 4.1134 - val_precision: 0.0560 - val_recall: 0.2413
Epoch 15/25
312/312 [==============================] - 1s 4ms/step - loss: 4.6336 - precision: 0.0454 - recall: 0.2397 - val_loss: 4.5869 - val_precision: 0.0467 - val_recall: 0.2240
Epoch 16/25
312/312 [==============================] - 1s 3ms/step - loss: 4.4598 - precision: 0.0492 - recall: 0.2509 - val_loss: 4.2531 - val_precision: 0.0524 - val_recall: 0.2326
Epoch 17/25
312/312 [==============================] - 1s 4ms/step - loss: 4.1858 - precision: 0.0506 - recall: 0.2397 - val_loss: 3.8781 - val_precision: 0.0465 - val_recall: 0.1823
Epoch 18/25
312/312 [==============================] - 1s 4ms/step - loss: 3.8732 - precision: 0.0405 - recall: 0.1704 - val_loss: 3.6943 - val_precision: 0.0454 - val_recall: 0.1667
Epoch 19/25
312/312 [==============================] - 1s 4ms/step - loss: 3.4705 - precision: 0.0396 - recall: 0.1461 - val_loss: 3.2010 - val_precision: 0.0483 - val_recall: 0.1476
Epoch 20/25
312/312 [==============================] - 1s 4ms/step - loss: 3.0757 - precision: 0.0333 - recall: 0.1030 - val_loss: 2.7556 - val_precision: 0.0308 - val_recall: 0.0747
Epoch 21/25
312/312 [==============================] - 1s 4ms/step - loss: 2.6314 - precision: 0.0264 - recall: 0.0655 - val_loss: 2.3652 - val_precision: 0.0317 - val_recall: 0.0608
Epoch 22/25
312/312 [==============================] - 1s 3ms/step - loss: 2.0667 - precision: 0.0273 - recall: 0.0487 - val_loss: 3.3379 - val_precision: 0.0246 - val_recall: 0.0747
Epoch 23/25
312/312 [==============================] - 1s 4ms/step - loss: 3.1269 - precision: 0.0190 - recall: 0.0581 - val_loss: 3.1142 - val_precision: 0.0233 - val_recall: 0.0642
Epoch 24/25
312/312 [==============================] - 1s 3ms/step - loss: 2.7097 - precision: 0.0193 - recall: 0.0487 - val_loss: 2.5656 - val_precision: 0.0221 - val_recall: 0.0469
Epoch 25/25
312/312 [==============================] - 1s 3ms/step - loss: 2.3790 - precision: 0.0227 - recall: 0.0468 - val_loss: 2.3607 - val_precision: 0.0235 - val_recall: 0.0434
In [51]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [52]:
output = model_performance(model3, X_test, y_test)
output

156/156 [==============================] - 0s 1ms/step
Out[52]:
Accuracy Recall Precision F1 Score
0 0.844257 0.844257 0.888372 0.865524
In [53]:
results.loc[3] = ['-','-','-',epochs,batch_size,'-','-','-',"SGD-LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                  ,output['Precision'][0],output['Recall'][0]]

results
Out[53]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
3 - - - 25 32 - - - SGD-LR(0.01) 27.868973 2.379025 2.360701 0.022727 0.023452 0.046816 0.043403 0.888372 0.844257
Model 4¶
  1. one Hidden layer - ReLu activation function - 80 nodes
  2. SGD with learning rate - 0.01
  3. Loss function - Binary Cross entropy
  4. Epochs - 25
  5. Batch size - 32
  6. Class weights taken into account
  7. Output layer - Sigmoid Activaion Function
In [54]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model4 = Sequential()
model4.add(Dense(80,activation="relu",input_dim=X_train.shape[1]))
model4.add(Dense(1,activation="sigmoid"))
model4.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 80)                3280      
                                                                 
 dense_1 (Dense)             (None, 1)                 81        
                                                                 
=================================================================
Total params: 3361 (13.13 KB)
Trainable params: 3361 (13.13 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [55]:
#Add the necessary configurations and fit the model

optimizer = keras.optimizers.SGD(learning_rate=0.01)
model4.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model4.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()

Epoch 1/25
312/312 [==============================] - 2s 4ms/step - loss: 0.4483 - precision: 0.1280 - recall: 0.4261 - val_loss: 0.3979 - val_precision: 0.2378 - val_recall: 0.8889
Epoch 2/25
312/312 [==============================] - 1s 3ms/step - loss: 0.3613 - precision: 0.2605 - recall: 0.8727 - val_loss: 0.3351 - val_precision: 0.3088 - val_recall: 0.9028
Epoch 3/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3284 - precision: 0.3142 - recall: 0.8839 - val_loss: 0.2721 - val_precision: 0.4046 - val_recall: 0.9097
Epoch 4/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3068 - precision: 0.3636 - recall: 0.8858 - val_loss: 0.2523 - val_precision: 0.4454 - val_recall: 0.9062
Epoch 5/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2905 - precision: 0.4041 - recall: 0.8764 - val_loss: 0.2433 - val_precision: 0.4815 - val_recall: 0.9062
Epoch 6/25
312/312 [==============================] - 1s 3ms/step - loss: 0.2775 - precision: 0.4225 - recall: 0.8876 - val_loss: 0.2304 - val_precision: 0.5188 - val_recall: 0.9080
Epoch 7/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2670 - precision: 0.4606 - recall: 0.8876 - val_loss: 0.2185 - val_precision: 0.5365 - val_recall: 0.9062
Epoch 8/25
312/312 [==============================] - 1s 3ms/step - loss: 0.2576 - precision: 0.4887 - recall: 0.8876 - val_loss: 0.2414 - val_precision: 0.4726 - val_recall: 0.9132
Epoch 9/25
312/312 [==============================] - 1s 3ms/step - loss: 0.2505 - precision: 0.4823 - recall: 0.8914 - val_loss: 0.1990 - val_precision: 0.6118 - val_recall: 0.9028
Epoch 10/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2448 - precision: 0.5197 - recall: 0.8895 - val_loss: 0.2009 - val_precision: 0.5794 - val_recall: 0.9062
Epoch 11/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2394 - precision: 0.5231 - recall: 0.8895 - val_loss: 0.1893 - val_precision: 0.6159 - val_recall: 0.8993
Epoch 12/25
312/312 [==============================] - 1s 3ms/step - loss: 0.2345 - precision: 0.5326 - recall: 0.8876 - val_loss: 0.1843 - val_precision: 0.6330 - val_recall: 0.8924
Epoch 13/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2302 - precision: 0.5570 - recall: 0.8876 - val_loss: 0.1935 - val_precision: 0.6103 - val_recall: 0.9028
Epoch 14/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2258 - precision: 0.5548 - recall: 0.8914 - val_loss: 0.1991 - val_precision: 0.5763 - val_recall: 0.9045
Epoch 15/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2212 - precision: 0.5504 - recall: 0.8895 - val_loss: 0.1776 - val_precision: 0.6318 - val_recall: 0.9028
Epoch 16/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2171 - precision: 0.5633 - recall: 0.8914 - val_loss: 0.1631 - val_precision: 0.6759 - val_recall: 0.8906
Epoch 17/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2161 - precision: 0.5812 - recall: 0.8914 - val_loss: 0.1894 - val_precision: 0.5957 - val_recall: 0.9080
Epoch 18/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2110 - precision: 0.5962 - recall: 0.8933 - val_loss: 0.2191 - val_precision: 0.4916 - val_recall: 0.9097
Epoch 19/25
312/312 [==============================] - 1s 3ms/step - loss: 0.2080 - precision: 0.5803 - recall: 0.8933 - val_loss: 0.2003 - val_precision: 0.5349 - val_recall: 0.9045
Epoch 20/25
312/312 [==============================] - 1s 3ms/step - loss: 0.2064 - precision: 0.5770 - recall: 0.8914 - val_loss: 0.1758 - val_precision: 0.6225 - val_recall: 0.9045
Epoch 21/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2035 - precision: 0.6028 - recall: 0.8951 - val_loss: 0.1727 - val_precision: 0.6283 - val_recall: 0.9010
Epoch 22/25
312/312 [==============================] - 1s 3ms/step - loss: 0.2004 - precision: 0.6003 - recall: 0.8970 - val_loss: 0.1820 - val_precision: 0.6002 - val_recall: 0.9045
Epoch 23/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1990 - precision: 0.6020 - recall: 0.9007 - val_loss: 0.1734 - val_precision: 0.6311 - val_recall: 0.9028
Epoch 24/25
312/312 [==============================] - 1s 3ms/step - loss: 0.1951 - precision: 0.5877 - recall: 0.8970 - val_loss: 0.1471 - val_precision: 0.7102 - val_recall: 0.8976
Epoch 25/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1938 - precision: 0.6089 - recall: 0.9007 - val_loss: 0.1448 - val_precision: 0.7077 - val_recall: 0.8993
In [56]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [57]:
output = model_performance(model4, X_test, y_test)
output

156/156 [==============================] - 0s 1ms/step
Out[57]:
Accuracy Recall Precision F1 Score
0 0.970134 0.970134 0.973761 0.971494
In [58]:
results.loc[4] = ['1','80','relu',epochs,batch_size,'-','-','-',"SGD_LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                  ,output['Precision'][0],output['Recall'][0]]

results
Out[58]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
3 - - - 25 32 - - - SGD-LR(0.01) 27.868973 2.379025 2.360701 0.022727 0.023452 0.046816 0.043403 0.888372 0.844257
4 1 80 relu 25 32 - - - SGD_LR(0.01) 41.567311 0.193805 0.144769 0.608861 0.707650 0.900749 0.899306 0.973761 0.970134
Model 5¶
  1. Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
  2. SGD with learning rate - 0.01
  3. Loss function - Binary Cross entropy
  4. Epochs - 25
  5. Batch size - 32
  6. Class weights taken into account
  7. Output layer - Sigmoid Activaion Function
In [59]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model5 = Sequential()
model5.add(Dense(80,activation="relu",input_dim=X_train.shape[1]))
model5.add(Dense(160,activation="relu"))
model5.add(Dense(1,activation="sigmoid"))
model5.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 80)                3280      
                                                                 
 dense_1 (Dense)             (None, 160)               12960     
                                                                 
 dense_2 (Dense)             (None, 1)                 161       
                                                                 
=================================================================
Total params: 16401 (64.07 KB)
Trainable params: 16401 (64.07 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [60]:
#Add the necessary configurations and fit the model

optimizer = keras.optimizers.SGD(learning_rate=0.01)
model5.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model5.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()

Epoch 1/25
312/312 [==============================] - 3s 6ms/step - loss: 0.4797 - precision: 0.2748 - recall: 0.8523 - val_loss: 0.4100 - val_precision: 0.2544 - val_recall: 0.8715
Epoch 2/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3524 - precision: 0.2733 - recall: 0.8633 - val_loss: 0.3081 - val_precision: 0.3529 - val_recall: 0.8854
Epoch 3/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3108 - precision: 0.3436 - recall: 0.8801 - val_loss: 0.2498 - val_precision: 0.4376 - val_recall: 0.8941
Epoch 4/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2837 - precision: 0.4012 - recall: 0.8858 - val_loss: 0.2195 - val_precision: 0.5010 - val_recall: 0.8906
Epoch 5/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2648 - precision: 0.4500 - recall: 0.8839 - val_loss: 0.2139 - val_precision: 0.5321 - val_recall: 0.9062
Epoch 6/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2475 - precision: 0.4857 - recall: 0.8933 - val_loss: 0.2074 - val_precision: 0.5617 - val_recall: 0.9010
Epoch 7/25
312/312 [==============================] - 1s 3ms/step - loss: 0.2348 - precision: 0.5175 - recall: 0.8876 - val_loss: 0.1834 - val_precision: 0.6061 - val_recall: 0.9028
Epoch 8/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2229 - precision: 0.5711 - recall: 0.8951 - val_loss: 0.2391 - val_precision: 0.4709 - val_recall: 0.9132
Epoch 9/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2131 - precision: 0.5772 - recall: 0.9026 - val_loss: 0.1525 - val_precision: 0.7467 - val_recall: 0.8854
Epoch 10/25
312/312 [==============================] - 1s 3ms/step - loss: 0.2063 - precision: 0.6299 - recall: 0.8989 - val_loss: 0.1570 - val_precision: 0.6803 - val_recall: 0.8976
Epoch 11/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1990 - precision: 0.6343 - recall: 0.8933 - val_loss: 0.1486 - val_precision: 0.7269 - val_recall: 0.8872
Epoch 12/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1922 - precision: 0.6639 - recall: 0.8951 - val_loss: 0.1518 - val_precision: 0.7377 - val_recall: 0.8837
Epoch 13/25
312/312 [==============================] - 1s 5ms/step - loss: 0.1861 - precision: 0.6751 - recall: 0.8989 - val_loss: 0.1591 - val_precision: 0.7007 - val_recall: 0.8941
Epoch 14/25
312/312 [==============================] - 1s 3ms/step - loss: 0.1809 - precision: 0.6823 - recall: 0.8970 - val_loss: 0.1760 - val_precision: 0.6157 - val_recall: 0.9010
Epoch 15/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1740 - precision: 0.6751 - recall: 0.9026 - val_loss: 0.1258 - val_precision: 0.7731 - val_recall: 0.8872
Epoch 16/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1695 - precision: 0.7022 - recall: 0.9007 - val_loss: 0.1241 - val_precision: 0.7703 - val_recall: 0.8906
Epoch 17/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1687 - precision: 0.7074 - recall: 0.9007 - val_loss: 0.1689 - val_precision: 0.6356 - val_recall: 0.8993
Epoch 18/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1622 - precision: 0.7198 - recall: 0.9045 - val_loss: 0.2079 - val_precision: 0.5246 - val_recall: 0.9080
Epoch 19/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1575 - precision: 0.7101 - recall: 0.9082 - val_loss: 0.1897 - val_precision: 0.5471 - val_recall: 0.9080
Epoch 20/25
312/312 [==============================] - 1s 3ms/step - loss: 0.1564 - precision: 0.6931 - recall: 0.9007 - val_loss: 0.1271 - val_precision: 0.7633 - val_recall: 0.8958
Epoch 21/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1529 - precision: 0.7192 - recall: 0.9064 - val_loss: 0.1332 - val_precision: 0.7344 - val_recall: 0.8976
Epoch 22/25
312/312 [==============================] - 1s 3ms/step - loss: 0.1485 - precision: 0.7162 - recall: 0.9026 - val_loss: 0.1360 - val_precision: 0.7161 - val_recall: 0.8976
Epoch 23/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1461 - precision: 0.7132 - recall: 0.9082 - val_loss: 0.1235 - val_precision: 0.7475 - val_recall: 0.8941
Epoch 24/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1403 - precision: 0.7417 - recall: 0.9195 - val_loss: 0.1136 - val_precision: 0.7795 - val_recall: 0.8958
Epoch 25/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1396 - precision: 0.7141 - recall: 0.9120 - val_loss: 0.1157 - val_precision: 0.7674 - val_recall: 0.8993
In [61]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [62]:
output = model_performance(model5, X_test, y_test)
output

156/156 [==============================] - 0s 2ms/step
Out[62]:
Accuracy Recall Precision F1 Score
0 0.977551 0.977551 0.97848 0.977946
In [63]:
results.loc[5] = ['2','80,160','relu,relu',epochs,batch_size,'-','-','-',"SGD_LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                  ,output['Precision'][0],output['Recall'][0]]

results
Out[63]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
3 - - - 25 32 - - - SGD-LR(0.01) 27.868973 2.379025 2.360701 0.022727 0.023452 0.046816 0.043403 0.888372 0.844257
4 1 80 relu 25 32 - - - SGD_LR(0.01) 41.567311 0.193805 0.144769 0.608861 0.707650 0.900749 0.899306 0.973761 0.970134
5 2 80,160 relu,relu 25 32 - - - SGD_LR(0.01) 41.894305 0.139635 0.115674 0.714076 0.767407 0.911985 0.899306 0.978480 0.977551
Model 6¶
  1. Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
  2. Adam Optimizer - learning_rate=0.01
  3. Loss function - Binary Cross entropy
  4. Epochs - 25
  5. Batch size - 32
  6. Class weights taken into account
  7. Output layer - Sigmoid Activaion Function
In [64]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model6 = Sequential()
model6.add(Dense(80,activation="relu",input_dim=X_train.shape[1]))
model6.add(Dense(160,activation="relu"))
model6.add(Dense(1,activation="sigmoid"))
model6.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 80)                3280      
                                                                 
 dense_1 (Dense)             (None, 160)               12960     
                                                                 
 dense_2 (Dense)             (None, 1)                 161       
                                                                 
=================================================================
Total params: 16401 (64.07 KB)
Trainable params: 16401 (64.07 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [65]:
#Add the necessary configurations and fit the model

adam_optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
model6.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model6.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()

Epoch 1/25
312/312 [==============================] - 2s 5ms/step - loss: 0.4154 - precision: 0.4503 - recall: 0.8685 - val_loss: 0.3027 - val_precision: 0.4393 - val_recall: 0.8854
Epoch 2/25
312/312 [==============================] - 1s 3ms/step - loss: 0.3012 - precision: 0.5477 - recall: 0.8708 - val_loss: 0.2881 - val_precision: 0.5471 - val_recall: 0.8767
Epoch 3/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2393 - precision: 0.6182 - recall: 0.8764 - val_loss: 0.0978 - val_precision: 0.9396 - val_recall: 0.8646
Epoch 4/25
312/312 [==============================] - 1s 3ms/step - loss: 0.2321 - precision: 0.7253 - recall: 0.8951 - val_loss: 0.0745 - val_precision: 0.9221 - val_recall: 0.8837
Epoch 5/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2299 - precision: 0.6547 - recall: 0.8914 - val_loss: 0.1423 - val_precision: 0.8907 - val_recall: 0.8628
Epoch 6/25
312/312 [==============================] - 1s 3ms/step - loss: 0.2875 - precision: 0.5972 - recall: 0.8745 - val_loss: 0.1715 - val_precision: 0.8491 - val_recall: 0.8594
Epoch 7/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2402 - precision: 0.6955 - recall: 0.8895 - val_loss: 0.1320 - val_precision: 0.8545 - val_recall: 0.8767
Epoch 8/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2696 - precision: 0.6389 - recall: 0.8914 - val_loss: 0.3039 - val_precision: 0.4856 - val_recall: 0.9097
Epoch 9/25
312/312 [==============================] - 2s 5ms/step - loss: 0.1807 - precision: 0.7642 - recall: 0.9045 - val_loss: 0.1232 - val_precision: 0.8946 - val_recall: 0.8837
Epoch 10/25
312/312 [==============================] - 2s 5ms/step - loss: 0.1820 - precision: 0.7606 - recall: 0.9045 - val_loss: 0.1354 - val_precision: 0.8304 - val_recall: 0.8924
Epoch 11/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1609 - precision: 0.8036 - recall: 0.9120 - val_loss: 0.1040 - val_precision: 0.9292 - val_recall: 0.8889
Epoch 12/25
312/312 [==============================] - 1s 3ms/step - loss: 0.1989 - precision: 0.7266 - recall: 0.9007 - val_loss: 0.1279 - val_precision: 0.8690 - val_recall: 0.8524
Epoch 13/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2516 - precision: 0.6403 - recall: 0.8933 - val_loss: 0.1660 - val_precision: 0.7896 - val_recall: 0.8924
Epoch 14/25
312/312 [==============================] - 1s 3ms/step - loss: 0.1696 - precision: 0.8039 - recall: 0.9213 - val_loss: 0.1146 - val_precision: 0.9204 - val_recall: 0.8837
Epoch 15/25
312/312 [==============================] - 1s 3ms/step - loss: 0.1730 - precision: 0.7810 - recall: 0.9082 - val_loss: 0.1406 - val_precision: 0.8955 - val_recall: 0.8628
Epoch 16/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1788 - precision: 0.6957 - recall: 0.9120 - val_loss: 0.1523 - val_precision: 0.7936 - val_recall: 0.9010
Epoch 17/25
312/312 [==============================] - 1s 3ms/step - loss: 0.1597 - precision: 0.7945 - recall: 0.9120 - val_loss: 0.1987 - val_precision: 0.5774 - val_recall: 0.8872
Epoch 18/25
312/312 [==============================] - 1s 3ms/step - loss: 0.1487 - precision: 0.8414 - recall: 0.9139 - val_loss: 0.1553 - val_precision: 0.7648 - val_recall: 0.8976
Epoch 19/25
312/312 [==============================] - 1s 3ms/step - loss: 0.1400 - precision: 0.8584 - recall: 0.9195 - val_loss: 0.1304 - val_precision: 0.9036 - val_recall: 0.8785
Epoch 20/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1509 - precision: 0.8180 - recall: 0.9176 - val_loss: 0.1470 - val_precision: 0.8251 - val_recall: 0.8681
Epoch 21/25
312/312 [==============================] - 2s 5ms/step - loss: 0.1792 - precision: 0.7598 - recall: 0.9064 - val_loss: 0.1607 - val_precision: 0.8298 - val_recall: 0.8889
Epoch 22/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1817 - precision: 0.7465 - recall: 0.9101 - val_loss: 0.1472 - val_precision: 0.8082 - val_recall: 0.8924
Epoch 23/25
312/312 [==============================] - 1s 3ms/step - loss: 0.1742 - precision: 0.7614 - recall: 0.9082 - val_loss: 0.1450 - val_precision: 0.7972 - val_recall: 0.8802
Epoch 24/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1692 - precision: 0.8088 - recall: 0.9270 - val_loss: 0.1375 - val_precision: 0.8055 - val_recall: 0.8698
Epoch 25/25
312/312 [==============================] - 1s 4ms/step - loss: 0.1451 - precision: 0.7850 - recall: 0.9232 - val_loss: 0.1335 - val_precision: 0.8403 - val_recall: 0.8767
In [66]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [67]:
output = model_performance(model6, X_test, y_test)
output

156/156 [==============================] - 0s 1ms/step
Out[67]:
Accuracy Recall Precision F1 Score
0 0.98176 0.98176 0.982125 0.981924
In [68]:
results.loc[6] = ['2','80,160','relu,relu',epochs,batch_size,'-','-','-',"Adam_LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                  ,output['Precision'][0],output['Recall'][0]]

results
Out[68]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
3 - - - 25 32 - - - SGD-LR(0.01) 27.868973 2.379025 2.360701 0.022727 0.023452 0.046816 0.043403 0.888372 0.844257
4 1 80 relu 25 32 - - - SGD_LR(0.01) 41.567311 0.193805 0.144769 0.608861 0.707650 0.900749 0.899306 0.973761 0.970134
5 2 80,160 relu,relu 25 32 - - - SGD_LR(0.01) 41.894305 0.139635 0.115674 0.714076 0.767407 0.911985 0.899306 0.978480 0.977551
6 2 80,160 relu,relu 25 32 - - - Adam_LR(0.01) 42.051322 0.145135 0.133539 0.785032 0.840266 0.923221 0.876736 0.982125 0.981760
Model 7¶
  1. Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
  2. Adam Optimizer - learning_rate = 1e-5
  3. Loss function - Binary Cross entropy
  4. Epochs - 25
  5. Batch size - 32
  6. Class weights taken into account
  7. Output layer - Sigmoid Activaion Function
In [69]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model7 = Sequential()
model7.add(Dense(80,activation="relu",input_dim=X_train.shape[1]))
model7.add(Dense(160,activation="relu"))
model7.add(Dense(1,activation="sigmoid"))
model7.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 80)                3280      
                                                                 
 dense_1 (Dense)             (None, 160)               12960     
                                                                 
 dense_2 (Dense)             (None, 1)                 161       
                                                                 
=================================================================
Total params: 16401 (64.07 KB)
Trainable params: 16401 (64.07 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [70]:
#Add the necessary configurations and fit the model

adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model7.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model7.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()

Epoch 1/25
312/312 [==============================] - 2s 4ms/step - loss: 0.6800 - precision: 0.1310 - recall: 0.8162 - val_loss: 0.7437 - val_precision: 0.0772 - val_recall: 0.8229
Epoch 2/25
312/312 [==============================] - 1s 3ms/step - loss: 0.6111 - precision: 0.0810 - recall: 0.8596 - val_loss: 0.6941 - val_precision: 0.0943 - val_recall: 0.8733
Epoch 3/25
312/312 [==============================] - 1s 3ms/step - loss: 0.5609 - precision: 0.0976 - recall: 0.8876 - val_loss: 0.6429 - val_precision: 0.1190 - val_recall: 0.8750
Epoch 4/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5231 - precision: 0.1218 - recall: 0.8745 - val_loss: 0.5984 - val_precision: 0.1495 - val_recall: 0.8628
Epoch 5/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4934 - precision: 0.1468 - recall: 0.8689 - val_loss: 0.5612 - val_precision: 0.1705 - val_recall: 0.8594
Epoch 6/25
312/312 [==============================] - 2s 5ms/step - loss: 0.4699 - precision: 0.1650 - recall: 0.8689 - val_loss: 0.5296 - val_precision: 0.1881 - val_recall: 0.8646
Epoch 7/25
312/312 [==============================] - 2s 5ms/step - loss: 0.4510 - precision: 0.1781 - recall: 0.8652 - val_loss: 0.5022 - val_precision: 0.2011 - val_recall: 0.8681
Epoch 8/25
312/312 [==============================] - 1s 3ms/step - loss: 0.4352 - precision: 0.1919 - recall: 0.8633 - val_loss: 0.4808 - val_precision: 0.2103 - val_recall: 0.8698
Epoch 9/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4219 - precision: 0.1984 - recall: 0.8689 - val_loss: 0.4627 - val_precision: 0.2197 - val_recall: 0.8715
Epoch 10/25
312/312 [==============================] - 1s 3ms/step - loss: 0.4105 - precision: 0.2083 - recall: 0.8727 - val_loss: 0.4447 - val_precision: 0.2308 - val_recall: 0.8750
Epoch 11/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4004 - precision: 0.2157 - recall: 0.8727 - val_loss: 0.4285 - val_precision: 0.2388 - val_recall: 0.8715
Epoch 12/25
312/312 [==============================] - 1s 3ms/step - loss: 0.3914 - precision: 0.2230 - recall: 0.8708 - val_loss: 0.4153 - val_precision: 0.2471 - val_recall: 0.8750
Epoch 13/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3834 - precision: 0.2304 - recall: 0.8745 - val_loss: 0.4037 - val_precision: 0.2528 - val_recall: 0.8750
Epoch 14/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3760 - precision: 0.2381 - recall: 0.8745 - val_loss: 0.3921 - val_precision: 0.2597 - val_recall: 0.8802
Epoch 15/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3694 - precision: 0.2420 - recall: 0.8783 - val_loss: 0.3833 - val_precision: 0.2650 - val_recall: 0.8819
Epoch 16/25
312/312 [==============================] - 1s 5ms/step - loss: 0.3631 - precision: 0.2474 - recall: 0.8801 - val_loss: 0.3713 - val_precision: 0.2752 - val_recall: 0.8819
Epoch 17/25
312/312 [==============================] - 2s 5ms/step - loss: 0.3574 - precision: 0.2610 - recall: 0.8783 - val_loss: 0.3633 - val_precision: 0.2828 - val_recall: 0.8872
Epoch 18/25
312/312 [==============================] - 1s 5ms/step - loss: 0.3521 - precision: 0.2669 - recall: 0.8801 - val_loss: 0.3570 - val_precision: 0.2897 - val_recall: 0.8941
Epoch 19/25
312/312 [==============================] - 1s 3ms/step - loss: 0.3472 - precision: 0.2695 - recall: 0.8801 - val_loss: 0.3487 - val_precision: 0.2958 - val_recall: 0.8941
Epoch 20/25
312/312 [==============================] - 1s 3ms/step - loss: 0.3425 - precision: 0.2764 - recall: 0.8820 - val_loss: 0.3440 - val_precision: 0.2998 - val_recall: 0.8993
Epoch 21/25
312/312 [==============================] - 1s 3ms/step - loss: 0.3381 - precision: 0.2818 - recall: 0.8839 - val_loss: 0.3374 - val_precision: 0.3087 - val_recall: 0.9010
Epoch 22/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3340 - precision: 0.2874 - recall: 0.8820 - val_loss: 0.3305 - val_precision: 0.3163 - val_recall: 0.9010
Epoch 23/25
312/312 [==============================] - 1s 3ms/step - loss: 0.3301 - precision: 0.2940 - recall: 0.8820 - val_loss: 0.3250 - val_precision: 0.3220 - val_recall: 0.9010
Epoch 24/25
312/312 [==============================] - 1s 3ms/step - loss: 0.3264 - precision: 0.3002 - recall: 0.8876 - val_loss: 0.3192 - val_precision: 0.3297 - val_recall: 0.8993
Epoch 25/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3230 - precision: 0.3053 - recall: 0.8914 - val_loss: 0.3132 - val_precision: 0.3372 - val_recall: 0.9010
In [71]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [72]:
output = model_performance(model7, X_test, y_test)
output

156/156 [==============================] - 0s 1ms/step
Out[72]:
Accuracy Recall Precision F1 Score
0 0.898577 0.898577 0.953981 0.917995
In [73]:
results.loc[7] = ['2','80,160','relu,relu',epochs,batch_size,'-','-','-',"Adam_LR(le-5)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                  ,output['Precision'][0],output['Recall'][0]]

results
Out[73]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
3 - - - 25 32 - - - SGD-LR(0.01) 27.868973 2.379025 2.360701 0.022727 0.023452 0.046816 0.043403 0.888372 0.844257
4 1 80 relu 25 32 - - - SGD_LR(0.01) 41.567311 0.193805 0.144769 0.608861 0.707650 0.900749 0.899306 0.973761 0.970134
5 2 80,160 relu,relu 25 32 - - - SGD_LR(0.01) 41.894305 0.139635 0.115674 0.714076 0.767407 0.911985 0.899306 0.978480 0.977551
6 2 80,160 relu,relu 25 32 - - - Adam_LR(0.01) 42.051322 0.145135 0.133539 0.785032 0.840266 0.923221 0.876736 0.982125 0.981760
7 2 80,160 relu,relu 25 32 - - - Adam_LR(le-5) 31.487571 0.322984 0.313230 0.305324 0.337232 0.891386 0.901042 0.953981 0.898577
Model 8¶
  1. Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
  2. Adam Optimizer - learning_rate = 1e-5
  3. Loss function - Binary Cross entropy
  4. Epochs - 25
  5. Batch size - 32
  6. Class weights taken into account
  7. Output layer - Sigmoid Activaion Function
  8. Drop out - 0.3
In [74]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model8 = Sequential([
    Dense(80, activation='relu', input_shape=(X_train.shape[1],)),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(160, activation='relu'),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(1, activation='sigmoid')  # For binary classification
])
model8.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 80)                3280      
                                                                 
 dropout (Dropout)           (None, 80)                0         
                                                                 
 dense_1 (Dense)             (None, 160)               12960     
                                                                 
 dropout_1 (Dropout)         (None, 160)               0         
                                                                 
 dense_2 (Dense)             (None, 1)                 161       
                                                                 
=================================================================
Total params: 16401 (64.07 KB)
Trainable params: 16401 (64.07 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [75]:
#Add the necessary configurations and fit the model

adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model8.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model8.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()

Epoch 1/25
312/312 [==============================] - 3s 6ms/step - loss: 0.7853 - precision: 0.1638 - recall: 0.5477 - val_loss: 0.6004 - val_precision: 0.0592 - val_recall: 0.1354
Epoch 2/25
312/312 [==============================] - 2s 5ms/step - loss: 0.7380 - precision: 0.0628 - recall: 0.3165 - val_loss: 0.6149 - val_precision: 0.0877 - val_recall: 0.3264
Epoch 3/25
312/312 [==============================] - 1s 4ms/step - loss: 0.6912 - precision: 0.0761 - recall: 0.4401 - val_loss: 0.6224 - val_precision: 0.1154 - val_recall: 0.5556
Epoch 4/25
312/312 [==============================] - 1s 4ms/step - loss: 0.6641 - precision: 0.0843 - recall: 0.5356 - val_loss: 0.6250 - val_precision: 0.1253 - val_recall: 0.7014
Epoch 5/25
312/312 [==============================] - 1s 4ms/step - loss: 0.6353 - precision: 0.0949 - recall: 0.6236 - val_loss: 0.6221 - val_precision: 0.1339 - val_recall: 0.8038
Epoch 6/25
312/312 [==============================] - 1s 3ms/step - loss: 0.6229 - precision: 0.0948 - recall: 0.6255 - val_loss: 0.6194 - val_precision: 0.1370 - val_recall: 0.8472
Epoch 7/25
312/312 [==============================] - 1s 3ms/step - loss: 0.5947 - precision: 0.1022 - recall: 0.6910 - val_loss: 0.6112 - val_precision: 0.1412 - val_recall: 0.8681
Epoch 8/25
312/312 [==============================] - 1s 3ms/step - loss: 0.5726 - precision: 0.1145 - recall: 0.7528 - val_loss: 0.6006 - val_precision: 0.1460 - val_recall: 0.8750
Epoch 9/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5600 - precision: 0.1207 - recall: 0.7865 - val_loss: 0.5916 - val_precision: 0.1503 - val_recall: 0.8872
Epoch 10/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5448 - precision: 0.1264 - recall: 0.7978 - val_loss: 0.5800 - val_precision: 0.1546 - val_recall: 0.8889
Epoch 11/25
312/312 [==============================] - 2s 5ms/step - loss: 0.5312 - precision: 0.1314 - recall: 0.8221 - val_loss: 0.5673 - val_precision: 0.1600 - val_recall: 0.8906
Epoch 12/25
312/312 [==============================] - 2s 5ms/step - loss: 0.5257 - precision: 0.1360 - recall: 0.8240 - val_loss: 0.5579 - val_precision: 0.1631 - val_recall: 0.8924
Epoch 13/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5132 - precision: 0.1374 - recall: 0.8090 - val_loss: 0.5472 - val_precision: 0.1667 - val_recall: 0.8924
Epoch 14/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4953 - precision: 0.1465 - recall: 0.8296 - val_loss: 0.5347 - val_precision: 0.1721 - val_recall: 0.8941
Epoch 15/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4891 - precision: 0.1461 - recall: 0.8221 - val_loss: 0.5224 - val_precision: 0.1773 - val_recall: 0.8889
Epoch 16/25
312/312 [==============================] - 1s 3ms/step - loss: 0.4845 - precision: 0.1559 - recall: 0.8315 - val_loss: 0.5104 - val_precision: 0.1825 - val_recall: 0.8889
Epoch 17/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4729 - precision: 0.1630 - recall: 0.8352 - val_loss: 0.4988 - val_precision: 0.1875 - val_recall: 0.8872
Epoch 18/25
312/312 [==============================] - 1s 3ms/step - loss: 0.4729 - precision: 0.1639 - recall: 0.8333 - val_loss: 0.4930 - val_precision: 0.1894 - val_recall: 0.8889
Epoch 19/25
312/312 [==============================] - 1s 3ms/step - loss: 0.4564 - precision: 0.1716 - recall: 0.8483 - val_loss: 0.4839 - val_precision: 0.1944 - val_recall: 0.8906
Epoch 20/25
312/312 [==============================] - 1s 3ms/step - loss: 0.4639 - precision: 0.1694 - recall: 0.8446 - val_loss: 0.4789 - val_precision: 0.1962 - val_recall: 0.8941
Epoch 21/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4497 - precision: 0.1729 - recall: 0.8427 - val_loss: 0.4713 - val_precision: 0.2007 - val_recall: 0.8941
Epoch 22/25
312/312 [==============================] - 2s 5ms/step - loss: 0.4394 - precision: 0.1810 - recall: 0.8652 - val_loss: 0.4619 - val_precision: 0.2062 - val_recall: 0.8941
Epoch 23/25
312/312 [==============================] - 2s 5ms/step - loss: 0.4362 - precision: 0.1814 - recall: 0.8521 - val_loss: 0.4536 - val_precision: 0.2112 - val_recall: 0.8958
Epoch 24/25
312/312 [==============================] - 1s 3ms/step - loss: 0.4406 - precision: 0.1877 - recall: 0.8521 - val_loss: 0.4470 - val_precision: 0.2147 - val_recall: 0.8958
Epoch 25/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4252 - precision: 0.1880 - recall: 0.8483 - val_loss: 0.4401 - val_precision: 0.2189 - val_recall: 0.8958
In [76]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [77]:
output = model_performance(model8, X_test, y_test)
output

156/156 [==============================] - 0s 2ms/step
Out[77]:
Accuracy Recall Precision F1 Score
0 0.818 0.818 0.946604 0.863412
In [78]:
results.loc[8] = ['2','80,160','relu,relu',epochs,batch_size,'-','-','0.3',"Adam_LR(le-5)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                  ,output['Precision'][0],output['Recall'][0]]

results
Out[78]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
3 - - - 25 32 - - - SGD-LR(0.01) 27.868973 2.379025 2.360701 0.022727 0.023452 0.046816 0.043403 0.888372 0.844257
4 1 80 relu 25 32 - - - SGD_LR(0.01) 41.567311 0.193805 0.144769 0.608861 0.707650 0.900749 0.899306 0.973761 0.970134
5 2 80,160 relu,relu 25 32 - - - SGD_LR(0.01) 41.894305 0.139635 0.115674 0.714076 0.767407 0.911985 0.899306 0.978480 0.977551
6 2 80,160 relu,relu 25 32 - - - Adam_LR(0.01) 42.051322 0.145135 0.133539 0.785032 0.840266 0.923221 0.876736 0.982125 0.981760
7 2 80,160 relu,relu 25 32 - - - Adam_LR(le-5) 31.487571 0.322984 0.313230 0.305324 0.337232 0.891386 0.901042 0.953981 0.898577
8 2 80,160 relu,relu 25 32 - - 0.3 Adam_LR(le-5) 42.194058 0.425188 0.440071 0.187967 0.218922 0.848315 0.895833 0.946604 0.818000
Model 9¶
  1. Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
  2. Adam Optimizer - learning_rate = 1e-5
  3. Loss function - Binary Cross entropy
  4. Epochs - 25
  5. Batch size - 32
  6. Class weights taken into account
  7. Output layer - Sigmoid Activaion Function
  8. Drop out - 0.3
  9. Batch Normalization added
In [79]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model9 = Sequential([
    Dense(80, activation='relu', input_shape=(X_train.shape[1],)),
    BatchNormalization(),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(160, activation='relu'),
    BatchNormalization(),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(1, activation='sigmoid')  # For binary classification
])
model9.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 80)                3280      
                                                                 
 batch_normalization (Batch  (None, 80)                320       
 Normalization)                                                  
                                                                 
 dropout (Dropout)           (None, 80)                0         
                                                                 
 dense_1 (Dense)             (None, 160)               12960     
                                                                 
 batch_normalization_1 (Bat  (None, 160)               640       
 chNormalization)                                                
                                                                 
 dropout_1 (Dropout)         (None, 160)               0         
                                                                 
 dense_2 (Dense)             (None, 1)                 161       
                                                                 
=================================================================
Total params: 17361 (67.82 KB)
Trainable params: 16881 (65.94 KB)
Non-trainable params: 480 (1.88 KB)
_________________________________________________________________
In [80]:
#Add the necessary configurations and fit the model

adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model9.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model9.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()

Epoch 1/25
312/312 [==============================] - 3s 5ms/step - loss: 0.9218 - precision: 0.1141 - recall: 0.7595 - val_loss: 0.7801 - val_precision: 0.0820 - val_recall: 0.7031
Epoch 2/25
312/312 [==============================] - 2s 5ms/step - loss: 0.8392 - precision: 0.0732 - recall: 0.6873 - val_loss: 0.7897 - val_precision: 0.0940 - val_recall: 0.7847
Epoch 3/25
312/312 [==============================] - 2s 6ms/step - loss: 0.7696 - precision: 0.0799 - recall: 0.7322 - val_loss: 0.7539 - val_precision: 0.1054 - val_recall: 0.8281
Epoch 4/25
312/312 [==============================] - 3s 9ms/step - loss: 0.7185 - precision: 0.0837 - recall: 0.7715 - val_loss: 0.7420 - val_precision: 0.1118 - val_recall: 0.8663
Epoch 5/25
312/312 [==============================] - 2s 7ms/step - loss: 0.7302 - precision: 0.0850 - recall: 0.7509 - val_loss: 0.7191 - val_precision: 0.1195 - val_recall: 0.8819
Epoch 6/25
312/312 [==============================] - 2s 6ms/step - loss: 0.6561 - precision: 0.0945 - recall: 0.8146 - val_loss: 0.6921 - val_precision: 0.1264 - val_recall: 0.8958
Epoch 7/25
312/312 [==============================] - 2s 5ms/step - loss: 0.6192 - precision: 0.0976 - recall: 0.8390 - val_loss: 0.6704 - val_precision: 0.1338 - val_recall: 0.9045
Epoch 8/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5946 - precision: 0.1013 - recall: 0.8464 - val_loss: 0.6491 - val_precision: 0.1392 - val_recall: 0.9097
Epoch 9/25
312/312 [==============================] - 2s 7ms/step - loss: 0.6136 - precision: 0.1017 - recall: 0.8240 - val_loss: 0.6278 - val_precision: 0.1455 - val_recall: 0.9132
Epoch 10/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5825 - precision: 0.1052 - recall: 0.8483 - val_loss: 0.6016 - val_precision: 0.1556 - val_recall: 0.9184
Epoch 11/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5695 - precision: 0.1076 - recall: 0.8502 - val_loss: 0.5851 - val_precision: 0.1623 - val_recall: 0.9219
Epoch 12/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5742 - precision: 0.1089 - recall: 0.8390 - val_loss: 0.5732 - val_precision: 0.1677 - val_recall: 0.9236
Epoch 13/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5496 - precision: 0.1123 - recall: 0.8502 - val_loss: 0.5427 - val_precision: 0.1811 - val_recall: 0.9201
Epoch 14/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5355 - precision: 0.1197 - recall: 0.8745 - val_loss: 0.5373 - val_precision: 0.1849 - val_recall: 0.9201
Epoch 15/25
312/312 [==============================] - 2s 5ms/step - loss: 0.5240 - precision: 0.1200 - recall: 0.8764 - val_loss: 0.5236 - val_precision: 0.1922 - val_recall: 0.9201
Epoch 16/25
312/312 [==============================] - 2s 5ms/step - loss: 0.5240 - precision: 0.1203 - recall: 0.8483 - val_loss: 0.5100 - val_precision: 0.2000 - val_recall: 0.9236
Epoch 17/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5101 - precision: 0.1240 - recall: 0.8558 - val_loss: 0.4974 - val_precision: 0.2071 - val_recall: 0.9236
Epoch 18/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4940 - precision: 0.1289 - recall: 0.8652 - val_loss: 0.4796 - val_precision: 0.2161 - val_recall: 0.9219
Epoch 19/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4877 - precision: 0.1302 - recall: 0.8596 - val_loss: 0.4717 - val_precision: 0.2232 - val_recall: 0.9236
Epoch 20/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4737 - precision: 0.1364 - recall: 0.8839 - val_loss: 0.4522 - val_precision: 0.2362 - val_recall: 0.9253
Epoch 21/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4730 - precision: 0.1389 - recall: 0.8614 - val_loss: 0.4400 - val_precision: 0.2454 - val_recall: 0.9253
Epoch 22/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4645 - precision: 0.1423 - recall: 0.8783 - val_loss: 0.4435 - val_precision: 0.2437 - val_recall: 0.9271
Epoch 23/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4566 - precision: 0.1451 - recall: 0.8839 - val_loss: 0.4228 - val_precision: 0.2573 - val_recall: 0.9219
Epoch 24/25
312/312 [==============================] - 1s 5ms/step - loss: 0.4600 - precision: 0.1430 - recall: 0.8408 - val_loss: 0.4124 - val_precision: 0.2642 - val_recall: 0.9219
Epoch 25/25
312/312 [==============================] - 2s 6ms/step - loss: 0.4508 - precision: 0.1456 - recall: 0.8670 - val_loss: 0.4017 - val_precision: 0.2709 - val_recall: 0.9236
In [81]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [82]:
output = model_performance(model9, X_test, y_test)
output

156/156 [==============================] - 0s 1ms/step
Out[82]:
Accuracy Recall Precision F1 Score
0 0.854279 0.854279 0.951479 0.888202
In [83]:
results.loc[9] = ['2','80,160','relu,relu',epochs,batch_size,'yes','-','0.3',"Adam_LR(le-5)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                  ,output['Precision'][0],output['Recall'][0]]

results
Out[83]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
3 - - - 25 32 - - - SGD-LR(0.01) 27.868973 2.379025 2.360701 0.022727 0.023452 0.046816 0.043403 0.888372 0.844257
4 1 80 relu 25 32 - - - SGD_LR(0.01) 41.567311 0.193805 0.144769 0.608861 0.707650 0.900749 0.899306 0.973761 0.970134
5 2 80,160 relu,relu 25 32 - - - SGD_LR(0.01) 41.894305 0.139635 0.115674 0.714076 0.767407 0.911985 0.899306 0.978480 0.977551
6 2 80,160 relu,relu 25 32 - - - Adam_LR(0.01) 42.051322 0.145135 0.133539 0.785032 0.840266 0.923221 0.876736 0.982125 0.981760
7 2 80,160 relu,relu 25 32 - - - Adam_LR(le-5) 31.487571 0.322984 0.313230 0.305324 0.337232 0.891386 0.901042 0.953981 0.898577
8 2 80,160 relu,relu 25 32 - - 0.3 Adam_LR(le-5) 42.194058 0.425188 0.440071 0.187967 0.218922 0.848315 0.895833 0.946604 0.818000
9 2 80,160 relu,relu 25 32 yes - 0.3 Adam_LR(le-5) 42.708833 0.450809 0.401692 0.145643 0.270876 0.867041 0.923611 0.951479 0.854279
Model 10¶
  1. Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
  2. Adam Optimizer - learning_rate = 1e-5
  3. Loss function - Binary Cross entropy
  4. Epochs - 25
  5. Batch size - 32
  6. Class weights taken into account
  7. Output layer - Sigmoid Activaion Function
  8. Drop out - 0.3
  9. Batch Normalization added
  10. Weight Initialization done - He (HL1), Glorot (HL2), RandomNormal (Output)
In [84]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model10 = Sequential([
    Dense(80, activation='relu', kernel_initializer=HeNormal(), input_shape=(X_train.shape[1],)),
    BatchNormalization(),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(160, activation='relu', kernel_initializer=GlorotUniform()),
    BatchNormalization(),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(1, activation='sigmoid', kernel_initializer=RandomNormal())  # For binary classification
])
model10.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 80)                3280      
                                                                 
 batch_normalization (Batch  (None, 80)                320       
 Normalization)                                                  
                                                                 
 dropout (Dropout)           (None, 80)                0         
                                                                 
 dense_1 (Dense)             (None, 160)               12960     
                                                                 
 batch_normalization_1 (Bat  (None, 160)               640       
 chNormalization)                                                
                                                                 
 dropout_1 (Dropout)         (None, 160)               0         
                                                                 
 dense_2 (Dense)             (None, 1)                 161       
                                                                 
=================================================================
Total params: 17361 (67.82 KB)
Trainable params: 16881 (65.94 KB)
Non-trainable params: 480 (1.88 KB)
_________________________________________________________________
In [85]:
#Add the necessary configurations and fit the model

adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model10.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model10.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()

Epoch 1/25
312/312 [==============================] - 3s 5ms/step - loss: 0.7269 - precision: 0.1210 - recall: 0.7658 - val_loss: 0.7111 - val_precision: 0.0803 - val_recall: 0.7170
Epoch 2/25
312/312 [==============================] - 1s 4ms/step - loss: 0.6642 - precision: 0.0736 - recall: 0.6948 - val_loss: 0.7010 - val_precision: 0.0942 - val_recall: 0.8507
Epoch 3/25
312/312 [==============================] - 1s 5ms/step - loss: 0.6309 - precision: 0.0831 - recall: 0.7865 - val_loss: 0.6854 - val_precision: 0.1016 - val_recall: 0.8906
Epoch 4/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5977 - precision: 0.0886 - recall: 0.8221 - val_loss: 0.6756 - val_precision: 0.1084 - val_recall: 0.9045
Epoch 5/25
312/312 [==============================] - 2s 5ms/step - loss: 0.5854 - precision: 0.0906 - recall: 0.8258 - val_loss: 0.6626 - val_precision: 0.1170 - val_recall: 0.9184
Epoch 6/25
312/312 [==============================] - 2s 6ms/step - loss: 0.5702 - precision: 0.0942 - recall: 0.8390 - val_loss: 0.6531 - val_precision: 0.1230 - val_recall: 0.9253
Epoch 7/25
312/312 [==============================] - 2s 5ms/step - loss: 0.5496 - precision: 0.0971 - recall: 0.8577 - val_loss: 0.6412 - val_precision: 0.1297 - val_recall: 0.9253
Epoch 8/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5409 - precision: 0.1035 - recall: 0.8820 - val_loss: 0.6338 - val_precision: 0.1343 - val_recall: 0.9271
Epoch 9/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5313 - precision: 0.1037 - recall: 0.8708 - val_loss: 0.6244 - val_precision: 0.1398 - val_recall: 0.9288
Epoch 10/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5142 - precision: 0.1094 - recall: 0.8820 - val_loss: 0.6149 - val_precision: 0.1449 - val_recall: 0.9288
Epoch 11/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5080 - precision: 0.1133 - recall: 0.8951 - val_loss: 0.6114 - val_precision: 0.1478 - val_recall: 0.9288
Epoch 12/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4982 - precision: 0.1132 - recall: 0.8839 - val_loss: 0.6052 - val_precision: 0.1520 - val_recall: 0.9358
Epoch 13/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4979 - precision: 0.1177 - recall: 0.8839 - val_loss: 0.5900 - val_precision: 0.1604 - val_recall: 0.9340
Epoch 14/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4873 - precision: 0.1194 - recall: 0.8914 - val_loss: 0.5873 - val_precision: 0.1616 - val_recall: 0.9358
Epoch 15/25
312/312 [==============================] - 2s 5ms/step - loss: 0.4827 - precision: 0.1211 - recall: 0.8989 - val_loss: 0.5790 - val_precision: 0.1662 - val_recall: 0.9340
Epoch 16/25
312/312 [==============================] - 2s 8ms/step - loss: 0.4811 - precision: 0.1223 - recall: 0.8989 - val_loss: 0.5698 - val_precision: 0.1699 - val_recall: 0.9340
Epoch 17/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4690 - precision: 0.1251 - recall: 0.8933 - val_loss: 0.5629 - val_precision: 0.1747 - val_recall: 0.9340
Epoch 18/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4673 - precision: 0.1274 - recall: 0.8858 - val_loss: 0.5494 - val_precision: 0.1811 - val_recall: 0.9323
Epoch 19/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4571 - precision: 0.1324 - recall: 0.8914 - val_loss: 0.5408 - val_precision: 0.1861 - val_recall: 0.9323
Epoch 20/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4491 - precision: 0.1333 - recall: 0.9007 - val_loss: 0.5309 - val_precision: 0.1925 - val_recall: 0.9306
Epoch 21/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4522 - precision: 0.1341 - recall: 0.8933 - val_loss: 0.5210 - val_precision: 0.1993 - val_recall: 0.9323
Epoch 22/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4425 - precision: 0.1388 - recall: 0.8989 - val_loss: 0.5222 - val_precision: 0.1987 - val_recall: 0.9323
Epoch 23/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4430 - precision: 0.1403 - recall: 0.9045 - val_loss: 0.5150 - val_precision: 0.2032 - val_recall: 0.9306
Epoch 24/25
312/312 [==============================] - 1s 5ms/step - loss: 0.4377 - precision: 0.1420 - recall: 0.8858 - val_loss: 0.5036 - val_precision: 0.2111 - val_recall: 0.9288
Epoch 25/25
312/312 [==============================] - 2s 6ms/step - loss: 0.4369 - precision: 0.1449 - recall: 0.8951 - val_loss: 0.4944 - val_precision: 0.2176 - val_recall: 0.9271
In [86]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [87]:
output = model_performance(model10, X_test, y_test)
output

156/156 [==============================] - 0s 2ms/step
Out[87]:
Accuracy Recall Precision F1 Score
0 0.801764 0.801764 0.948019 0.852611
In [88]:
results.loc[10] = ['2','80,160','relu,relu',epochs,batch_size,'yes','he/glorot/randnorm','0.3',"Adam_LR(le-5)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                   ,output['Precision'][0],output['Recall'][0]]

results
Out[88]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
3 - - - 25 32 - - - SGD-LR(0.01) 27.868973 2.379025 2.360701 0.022727 0.023452 0.046816 0.043403 0.888372 0.844257
4 1 80 relu 25 32 - - - SGD_LR(0.01) 41.567311 0.193805 0.144769 0.608861 0.707650 0.900749 0.899306 0.973761 0.970134
5 2 80,160 relu,relu 25 32 - - - SGD_LR(0.01) 41.894305 0.139635 0.115674 0.714076 0.767407 0.911985 0.899306 0.978480 0.977551
6 2 80,160 relu,relu 25 32 - - - Adam_LR(0.01) 42.051322 0.145135 0.133539 0.785032 0.840266 0.923221 0.876736 0.982125 0.981760
7 2 80,160 relu,relu 25 32 - - - Adam_LR(le-5) 31.487571 0.322984 0.313230 0.305324 0.337232 0.891386 0.901042 0.953981 0.898577
8 2 80,160 relu,relu 25 32 - - 0.3 Adam_LR(le-5) 42.194058 0.425188 0.440071 0.187967 0.218922 0.848315 0.895833 0.946604 0.818000
9 2 80,160 relu,relu 25 32 yes - 0.3 Adam_LR(le-5) 42.708833 0.450809 0.401692 0.145643 0.270876 0.867041 0.923611 0.951479 0.854279
10 2 80,160 relu,relu 25 32 yes he/glorot/randnorm 0.3 Adam_LR(le-5) 37.227174 0.436856 0.494443 0.144936 0.217604 0.895131 0.927083 0.948019 0.801764
Model 11¶
  1. Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
  2. Adam Optimizer - learning_rate = 1e-5
  3. Loss function - Binary Cross entropy
  4. Epochs - 25
  5. Batch size - 32
  6. Class weights taken into account
  7. Output layer - Sigmoid Activaion Function
  8. Drop out - 0.3
  9. Batch Normalization added
  10. Weight Initialization done - He (HL1), Glorot (HL2), RandomNormal (Output)
  11. Regularization - L1 and L2
In [89]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model11 = Sequential([
    Dense(80, activation='relu', kernel_initializer=HeNormal(),kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4), input_shape=(X_train.shape[1],)),
    BatchNormalization(),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(160, activation='relu', kernel_initializer=GlorotUniform(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)),
    BatchNormalization(),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(1, activation='sigmoid', kernel_initializer=RandomNormal(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4))  # For binary classification
])
model11.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 80)                3280      
                                                                 
 batch_normalization (Batch  (None, 80)                320       
 Normalization)                                                  
                                                                 
 dropout (Dropout)           (None, 80)                0         
                                                                 
 dense_1 (Dense)             (None, 160)               12960     
                                                                 
 batch_normalization_1 (Bat  (None, 160)               640       
 chNormalization)                                                
                                                                 
 dropout_1 (Dropout)         (None, 160)               0         
                                                                 
 dense_2 (Dense)             (None, 1)                 161       
                                                                 
=================================================================
Total params: 17361 (67.82 KB)
Trainable params: 16881 (65.94 KB)
Non-trainable params: 480 (1.88 KB)
_________________________________________________________________
In [90]:
#Add the necessary configurations and fit the model

adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model11.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model11.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()

Epoch 1/25
312/312 [==============================] - 3s 6ms/step - loss: 0.7786 - precision: 0.1103 - recall: 0.7459 - val_loss: 0.7860 - val_precision: 0.0763 - val_recall: 0.7517
Epoch 2/25
312/312 [==============================] - 1s 4ms/step - loss: 0.7263 - precision: 0.0701 - recall: 0.6573 - val_loss: 0.7735 - val_precision: 0.0867 - val_recall: 0.8264
Epoch 3/25
312/312 [==============================] - 1s 5ms/step - loss: 0.6839 - precision: 0.0802 - recall: 0.7472 - val_loss: 0.7607 - val_precision: 0.0961 - val_recall: 0.8698
Epoch 4/25
312/312 [==============================] - 1s 4ms/step - loss: 0.6414 - precision: 0.0883 - recall: 0.7921 - val_loss: 0.7482 - val_precision: 0.1061 - val_recall: 0.8924
Epoch 5/25
312/312 [==============================] - 1s 5ms/step - loss: 0.6184 - precision: 0.0943 - recall: 0.8333 - val_loss: 0.7419 - val_precision: 0.1114 - val_recall: 0.9010
Epoch 6/25
312/312 [==============================] - 2s 7ms/step - loss: 0.5992 - precision: 0.1005 - recall: 0.8539 - val_loss: 0.7221 - val_precision: 0.1216 - val_recall: 0.9062
Epoch 7/25
312/312 [==============================] - 2s 6ms/step - loss: 0.5878 - precision: 0.1032 - recall: 0.8633 - val_loss: 0.7159 - val_precision: 0.1261 - val_recall: 0.9149
Epoch 8/25
312/312 [==============================] - 1s 5ms/step - loss: 0.5771 - precision: 0.1054 - recall: 0.8745 - val_loss: 0.7099 - val_precision: 0.1309 - val_recall: 0.9219
Epoch 9/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5561 - precision: 0.1115 - recall: 0.8895 - val_loss: 0.6992 - val_precision: 0.1378 - val_recall: 0.9253
Epoch 10/25
312/312 [==============================] - 1s 5ms/step - loss: 0.5564 - precision: 0.1116 - recall: 0.8801 - val_loss: 0.6869 - val_precision: 0.1439 - val_recall: 0.9271
Epoch 11/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5404 - precision: 0.1182 - recall: 0.8895 - val_loss: 0.6824 - val_precision: 0.1471 - val_recall: 0.9306
Epoch 12/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5326 - precision: 0.1176 - recall: 0.8764 - val_loss: 0.6763 - val_precision: 0.1504 - val_recall: 0.9340
Epoch 13/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5320 - precision: 0.1208 - recall: 0.8839 - val_loss: 0.6621 - val_precision: 0.1568 - val_recall: 0.9288
Epoch 14/25
312/312 [==============================] - 1s 5ms/step - loss: 0.5313 - precision: 0.1228 - recall: 0.8708 - val_loss: 0.6530 - val_precision: 0.1604 - val_recall: 0.9253
Epoch 15/25
312/312 [==============================] - 2s 5ms/step - loss: 0.5145 - precision: 0.1280 - recall: 0.9007 - val_loss: 0.6541 - val_precision: 0.1610 - val_recall: 0.9306
Epoch 16/25
312/312 [==============================] - 2s 6ms/step - loss: 0.5112 - precision: 0.1297 - recall: 0.8951 - val_loss: 0.6406 - val_precision: 0.1670 - val_recall: 0.9288
Epoch 17/25
312/312 [==============================] - 2s 5ms/step - loss: 0.5002 - precision: 0.1325 - recall: 0.9026 - val_loss: 0.6330 - val_precision: 0.1714 - val_recall: 0.9288
Epoch 18/25
312/312 [==============================] - 1s 4ms/step - loss: 0.5064 - precision: 0.1324 - recall: 0.8895 - val_loss: 0.6191 - val_precision: 0.1778 - val_recall: 0.9288
Epoch 19/25
312/312 [==============================] - 1s 5ms/step - loss: 0.4869 - precision: 0.1389 - recall: 0.9139 - val_loss: 0.6147 - val_precision: 0.1801 - val_recall: 0.9306
Epoch 20/25
312/312 [==============================] - 1s 5ms/step - loss: 0.4901 - precision: 0.1385 - recall: 0.8989 - val_loss: 0.5995 - val_precision: 0.1867 - val_recall: 0.9271
Epoch 21/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4845 - precision: 0.1403 - recall: 0.8876 - val_loss: 0.5976 - val_precision: 0.1897 - val_recall: 0.9306
Epoch 22/25
312/312 [==============================] - 1s 5ms/step - loss: 0.4734 - precision: 0.1475 - recall: 0.9082 - val_loss: 0.5916 - val_precision: 0.1939 - val_recall: 0.9306
Epoch 23/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4810 - precision: 0.1494 - recall: 0.9007 - val_loss: 0.5787 - val_precision: 0.2008 - val_recall: 0.9288
Epoch 24/25
312/312 [==============================] - 1s 5ms/step - loss: 0.4694 - precision: 0.1490 - recall: 0.8951 - val_loss: 0.5682 - val_precision: 0.2070 - val_recall: 0.9253
Epoch 25/25
312/312 [==============================] - 2s 6ms/step - loss: 0.4675 - precision: 0.1506 - recall: 0.8951 - val_loss: 0.5597 - val_precision: 0.2135 - val_recall: 0.9253
In [91]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [92]:
output = model_performance(model11, X_test, y_test)
print(output)
output_1 = model_performance(model11, X_train, y_train)
print(output_1)

156/156 [==============================] - 0s 1ms/step
   Accuracy    Recall  Precision  F1 Score
0  0.797354  0.797354   0.946944  0.849524
312/312 [==============================] - 0s 1ms/step
   Accuracy    Recall  Precision  F1 Score
0  0.794931  0.794931   0.951182  0.849408
In [93]:
results.loc[11] = ['2','80,160','relu,relu',epochs,batch_size,'yes','he/glorot/randnorm','0.3',"Adam_LR(le-5)/L1,L2",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                   ,output['Precision'][0],output['Recall'][0]]

results
Out[93]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
3 - - - 25 32 - - - SGD-LR(0.01) 27.868973 2.379025 2.360701 0.022727 0.023452 0.046816 0.043403 0.888372 0.844257
4 1 80 relu 25 32 - - - SGD_LR(0.01) 41.567311 0.193805 0.144769 0.608861 0.707650 0.900749 0.899306 0.973761 0.970134
5 2 80,160 relu,relu 25 32 - - - SGD_LR(0.01) 41.894305 0.139635 0.115674 0.714076 0.767407 0.911985 0.899306 0.978480 0.977551
6 2 80,160 relu,relu 25 32 - - - Adam_LR(0.01) 42.051322 0.145135 0.133539 0.785032 0.840266 0.923221 0.876736 0.982125 0.981760
7 2 80,160 relu,relu 25 32 - - - Adam_LR(le-5) 31.487571 0.322984 0.313230 0.305324 0.337232 0.891386 0.901042 0.953981 0.898577
8 2 80,160 relu,relu 25 32 - - 0.3 Adam_LR(le-5) 42.194058 0.425188 0.440071 0.187967 0.218922 0.848315 0.895833 0.946604 0.818000
9 2 80,160 relu,relu 25 32 yes - 0.3 Adam_LR(le-5) 42.708833 0.450809 0.401692 0.145643 0.270876 0.867041 0.923611 0.951479 0.854279
10 2 80,160 relu,relu 25 32 yes he/glorot/randnorm 0.3 Adam_LR(le-5) 37.227174 0.436856 0.494443 0.144936 0.217604 0.895131 0.927083 0.948019 0.801764
11 2 80,160 relu,relu 25 32 yes he/glorot/randnorm 0.3 Adam_LR(le-5)/L1,L2 42.635221 0.467466 0.559652 0.150646 0.213456 0.895131 0.925347 0.946944 0.797354
Model 12¶
  1. Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
  2. AdaGrad Optimizer - learning_rate = 0.01
  3. Loss function - Binary Cross entropy
  4. Epochs - 25
  5. Batch size - 32
  6. Class weights taken into account
  7. Output layer - Sigmoid Activaion Function
  8. Drop out - 0.3
  9. Batch Normalization added
  10. Weight Initialization done - He (HL1), Glorot (HL2), RandomNormal (Output)
  11. Regularization - L1 and L2
In [94]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model12 = Sequential([
    Dense(80, activation='relu', kernel_initializer=HeNormal(),kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4), input_shape=(X_train.shape[1],)),
    BatchNormalization(),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(160, activation='relu', kernel_initializer=GlorotUniform(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)),
    BatchNormalization(),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(1, activation='sigmoid', kernel_initializer=RandomNormal(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4))  # For binary classification
])
model12.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 80)                3280      
                                                                 
 batch_normalization (Batch  (None, 80)                320       
 Normalization)                                                  
                                                                 
 dropout (Dropout)           (None, 80)                0         
                                                                 
 dense_1 (Dense)             (None, 160)               12960     
                                                                 
 batch_normalization_1 (Bat  (None, 160)               640       
 chNormalization)                                                
                                                                 
 dropout_1 (Dropout)         (None, 160)               0         
                                                                 
 dense_2 (Dense)             (None, 1)                 161       
                                                                 
=================================================================
Total params: 17361 (67.82 KB)
Trainable params: 16881 (65.94 KB)
Non-trainable params: 480 (1.88 KB)
_________________________________________________________________
In [95]:
#Add the necessary configurations and fit the model

adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
# Customize AdaGrad optimizer
adagrad_optimizer = tf.keras.optimizers.Adagrad(learning_rate=0.01)
model12.compile(optimizer=adagrad_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model12.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()

Epoch 1/25
312/312 [==============================] - 3s 5ms/step - loss: 0.5063 - precision: 0.1732 - recall: 0.8964 - val_loss: 0.4447 - val_precision: 0.3110 - val_recall: 0.9028
Epoch 2/25
312/312 [==============================] - 1s 4ms/step - loss: 0.4198 - precision: 0.2149 - recall: 0.8727 - val_loss: 0.3382 - val_precision: 0.4240 - val_recall: 0.8958
Epoch 3/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3685 - precision: 0.2667 - recall: 0.8801 - val_loss: 0.2859 - val_precision: 0.5129 - val_recall: 0.8958
Epoch 4/25
312/312 [==============================] - 2s 8ms/step - loss: 0.3500 - precision: 0.3031 - recall: 0.8652 - val_loss: 0.2550 - val_precision: 0.5677 - val_recall: 0.8958
Epoch 5/25
312/312 [==============================] - 2s 5ms/step - loss: 0.3295 - precision: 0.3433 - recall: 0.8801 - val_loss: 0.2303 - val_precision: 0.6273 - val_recall: 0.8941
Epoch 6/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3162 - precision: 0.3628 - recall: 0.8764 - val_loss: 0.2117 - val_precision: 0.7004 - val_recall: 0.8889
Epoch 7/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3128 - precision: 0.3882 - recall: 0.8783 - val_loss: 0.2017 - val_precision: 0.7288 - val_recall: 0.8958
Epoch 8/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2917 - precision: 0.4247 - recall: 0.8876 - val_loss: 0.1934 - val_precision: 0.7059 - val_recall: 0.8958
Epoch 9/25
312/312 [==============================] - 1s 4ms/step - loss: 0.3079 - precision: 0.4173 - recall: 0.8783 - val_loss: 0.1826 - val_precision: 0.7773 - val_recall: 0.8906
Epoch 10/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2879 - precision: 0.4454 - recall: 0.8783 - val_loss: 0.1761 - val_precision: 0.7743 - val_recall: 0.8993
Epoch 11/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2986 - precision: 0.4215 - recall: 0.8801 - val_loss: 0.1747 - val_precision: 0.7806 - val_recall: 0.8958
Epoch 12/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2733 - precision: 0.4535 - recall: 0.8858 - val_loss: 0.1697 - val_precision: 0.8339 - val_recall: 0.8889
Epoch 13/25
312/312 [==============================] - 2s 6ms/step - loss: 0.2826 - precision: 0.4820 - recall: 0.8764 - val_loss: 0.1686 - val_precision: 0.8100 - val_recall: 0.8958
Epoch 14/25
312/312 [==============================] - 2s 8ms/step - loss: 0.2794 - precision: 0.4668 - recall: 0.8820 - val_loss: 0.1661 - val_precision: 0.7997 - val_recall: 0.9010
Epoch 15/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2689 - precision: 0.4645 - recall: 0.8933 - val_loss: 0.1645 - val_precision: 0.7997 - val_recall: 0.9010
Epoch 16/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2731 - precision: 0.4765 - recall: 0.8933 - val_loss: 0.1613 - val_precision: 0.8122 - val_recall: 0.9010
Epoch 17/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2683 - precision: 0.4917 - recall: 0.8876 - val_loss: 0.1601 - val_precision: 0.8173 - val_recall: 0.9010
Epoch 18/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2737 - precision: 0.4916 - recall: 0.8820 - val_loss: 0.1626 - val_precision: 0.7930 - val_recall: 0.9045
Epoch 19/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2730 - precision: 0.4840 - recall: 0.8764 - val_loss: 0.1585 - val_precision: 0.8201 - val_recall: 0.8941
Epoch 20/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2645 - precision: 0.4958 - recall: 0.8914 - val_loss: 0.1592 - val_precision: 0.7982 - val_recall: 0.9062
Epoch 21/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2552 - precision: 0.4958 - recall: 0.8914 - val_loss: 0.1544 - val_precision: 0.8264 - val_recall: 0.9010
Epoch 22/25
312/312 [==============================] - 1s 5ms/step - loss: 0.2543 - precision: 0.5081 - recall: 0.8858 - val_loss: 0.1593 - val_precision: 0.8278 - val_recall: 0.9010
Epoch 23/25
312/312 [==============================] - 2s 5ms/step - loss: 0.2605 - precision: 0.5080 - recall: 0.8895 - val_loss: 0.1548 - val_precision: 0.8296 - val_recall: 0.8958
Epoch 24/25
312/312 [==============================] - 2s 5ms/step - loss: 0.2514 - precision: 0.4990 - recall: 0.8951 - val_loss: 0.1514 - val_precision: 0.8464 - val_recall: 0.8993
Epoch 25/25
312/312 [==============================] - 1s 4ms/step - loss: 0.2562 - precision: 0.5283 - recall: 0.8914 - val_loss: 0.1508 - val_precision: 0.8296 - val_recall: 0.9045
In [96]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [97]:
output = model_performance(model12, X_test, y_test)
print(output)
output_1 = model_performance(model12, X_train, y_train)
print(output_1)

156/156 [==============================] - 0s 1ms/step
   Accuracy    Recall  Precision  F1 Score
0  0.980758  0.980758   0.981178  0.980946
312/312 [==============================] - 0s 1ms/step
   Accuracy   Recall  Precision  F1 Score
0   0.98317  0.98317   0.984289  0.983591
In [98]:
results.loc[12] = ['2','80,160','relu,relu',epochs,batch_size,'yes','he/glorot/randnorm','0.3',"Adagrad_LR(0.01)/L1,L2",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                   ,output['Precision'][0],output['Recall'][0]]

results
Out[98]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
3 - - - 25 32 - - - SGD-LR(0.01) 27.868973 2.379025 2.360701 0.022727 0.023452 0.046816 0.043403 0.888372 0.844257
4 1 80 relu 25 32 - - - SGD_LR(0.01) 41.567311 0.193805 0.144769 0.608861 0.707650 0.900749 0.899306 0.973761 0.970134
5 2 80,160 relu,relu 25 32 - - - SGD_LR(0.01) 41.894305 0.139635 0.115674 0.714076 0.767407 0.911985 0.899306 0.978480 0.977551
6 2 80,160 relu,relu 25 32 - - - Adam_LR(0.01) 42.051322 0.145135 0.133539 0.785032 0.840266 0.923221 0.876736 0.982125 0.981760
7 2 80,160 relu,relu 25 32 - - - Adam_LR(le-5) 31.487571 0.322984 0.313230 0.305324 0.337232 0.891386 0.901042 0.953981 0.898577
8 2 80,160 relu,relu 25 32 - - 0.3 Adam_LR(le-5) 42.194058 0.425188 0.440071 0.187967 0.218922 0.848315 0.895833 0.946604 0.818000
9 2 80,160 relu,relu 25 32 yes - 0.3 Adam_LR(le-5) 42.708833 0.450809 0.401692 0.145643 0.270876 0.867041 0.923611 0.951479 0.854279
10 2 80,160 relu,relu 25 32 yes he/glorot/randnorm 0.3 Adam_LR(le-5) 37.227174 0.436856 0.494443 0.144936 0.217604 0.895131 0.927083 0.948019 0.801764
11 2 80,160 relu,relu 25 32 yes he/glorot/randnorm 0.3 Adam_LR(le-5)/L1,L2 42.635221 0.467466 0.559652 0.150646 0.213456 0.895131 0.925347 0.946944 0.797354
12 2 80,160 relu,relu 25 32 yes he/glorot/randnorm 0.3 Adagrad_LR(0.01)/L1,L2 37.517705 0.256188 0.150753 0.528302 0.829618 0.891386 0.904514 0.981178 0.980758
Model 13 (Model 11 with epoch change)¶
  1. Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
  2. Adam Optimizer - learning_rate = 1e-5
  3. Loss function - Binary Cross entropy
  4. Epochs - 50
  5. Batch size - 32
  6. Class weights taken into account
  7. Output layer - Sigmoid Activaion Function
  8. Drop out - 0.3
  9. Batch Normalization added
  10. Weight Initialization done - He (HL1), Glorot (HL2), RandomNormal (Output)
  11. Regularization - L1 and L2
In [99]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

#Initializing the neural network
model13 = Sequential([
    Dense(80, activation='relu', kernel_initializer=HeNormal(),kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4), input_shape=(X_train.shape[1],)),
    BatchNormalization(),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(160, activation='relu', kernel_initializer=GlorotUniform(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)),
    BatchNormalization(),
    Dropout(0.3),  # Dropout layer with 30% dropout rate
    Dense(1, activation='sigmoid', kernel_initializer=RandomNormal(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4))  # For binary classification
])
model13.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 80)                3280      
                                                                 
 batch_normalization (Batch  (None, 80)                320       
 Normalization)                                                  
                                                                 
 dropout (Dropout)           (None, 80)                0         
                                                                 
 dense_1 (Dense)             (None, 160)               12960     
                                                                 
 batch_normalization_1 (Bat  (None, 160)               640       
 chNormalization)                                                
                                                                 
 dropout_1 (Dropout)         (None, 160)               0         
                                                                 
 dense_2 (Dense)             (None, 1)                 161       
                                                                 
=================================================================
Total params: 17361 (67.82 KB)
Trainable params: 16881 (65.94 KB)
Non-trainable params: 480 (1.88 KB)
_________________________________________________________________
In [100]:
#Add the necessary configurations and fit the model

adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model13.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 50
batch_size = 32
start = time.time()
history = model13.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()

Epoch 1/50
312/312 [==============================] - 3s 5ms/step - loss: 0.7501 - precision: 0.1531 - recall: 0.7432 - val_loss: 0.7247 - val_precision: 0.1000 - val_recall: 0.7743
Epoch 2/50
312/312 [==============================] - 1s 5ms/step - loss: 0.6894 - precision: 0.0792 - recall: 0.7022 - val_loss: 0.7073 - val_precision: 0.1127 - val_recall: 0.8438
Epoch 3/50
312/312 [==============================] - 1s 5ms/step - loss: 0.6539 - precision: 0.0890 - recall: 0.7715 - val_loss: 0.6970 - val_precision: 0.1222 - val_recall: 0.8837
Epoch 4/50
312/312 [==============================] - 2s 6ms/step - loss: 0.6177 - precision: 0.0930 - recall: 0.8015 - val_loss: 0.6867 - val_precision: 0.1295 - val_recall: 0.8976
Epoch 5/50
312/312 [==============================] - 2s 6ms/step - loss: 0.6159 - precision: 0.0959 - recall: 0.8109 - val_loss: 0.6764 - val_precision: 0.1375 - val_recall: 0.9149
Epoch 6/50
312/312 [==============================] - 2s 5ms/step - loss: 0.5910 - precision: 0.1048 - recall: 0.8446 - val_loss: 0.6666 - val_precision: 0.1446 - val_recall: 0.9236
Epoch 7/50
312/312 [==============================] - 1s 4ms/step - loss: 0.5733 - precision: 0.1065 - recall: 0.8558 - val_loss: 0.6611 - val_precision: 0.1494 - val_recall: 0.9236
Epoch 8/50
312/312 [==============================] - 1s 5ms/step - loss: 0.5597 - precision: 0.1106 - recall: 0.8614 - val_loss: 0.6573 - val_precision: 0.1531 - val_recall: 0.9306
Epoch 9/50
312/312 [==============================] - 1s 5ms/step - loss: 0.5498 - precision: 0.1142 - recall: 0.8820 - val_loss: 0.6501 - val_precision: 0.1581 - val_recall: 0.9323
Epoch 10/50
312/312 [==============================] - 1s 5ms/step - loss: 0.5454 - precision: 0.1151 - recall: 0.8820 - val_loss: 0.6346 - val_precision: 0.1660 - val_recall: 0.9288
Epoch 11/50
312/312 [==============================] - 1s 5ms/step - loss: 0.5373 - precision: 0.1188 - recall: 0.8801 - val_loss: 0.6308 - val_precision: 0.1705 - val_recall: 0.9323
Epoch 12/50
312/312 [==============================] - 1s 4ms/step - loss: 0.5262 - precision: 0.1237 - recall: 0.8914 - val_loss: 0.6257 - val_precision: 0.1741 - val_recall: 0.9340
Epoch 13/50
312/312 [==============================] - 2s 7ms/step - loss: 0.5145 - precision: 0.1281 - recall: 0.9101 - val_loss: 0.6171 - val_precision: 0.1781 - val_recall: 0.9358
Epoch 14/50
312/312 [==============================] - 2s 6ms/step - loss: 0.5129 - precision: 0.1288 - recall: 0.8876 - val_loss: 0.6050 - val_precision: 0.1848 - val_recall: 0.9358
Epoch 15/50
312/312 [==============================] - 1s 5ms/step - loss: 0.5025 - precision: 0.1315 - recall: 0.9064 - val_loss: 0.6070 - val_precision: 0.1840 - val_recall: 0.9340
Epoch 16/50
312/312 [==============================] - 1s 5ms/step - loss: 0.4974 - precision: 0.1374 - recall: 0.9157 - val_loss: 0.5913 - val_precision: 0.1912 - val_recall: 0.9340
Epoch 17/50
312/312 [==============================] - 1s 4ms/step - loss: 0.4973 - precision: 0.1359 - recall: 0.8914 - val_loss: 0.5870 - val_precision: 0.1948 - val_recall: 0.9340
Epoch 18/50
312/312 [==============================] - 1s 5ms/step - loss: 0.4922 - precision: 0.1385 - recall: 0.9007 - val_loss: 0.5784 - val_precision: 0.1999 - val_recall: 0.9340
Epoch 19/50
312/312 [==============================] - 1s 4ms/step - loss: 0.4822 - precision: 0.1414 - recall: 0.9007 - val_loss: 0.5686 - val_precision: 0.2068 - val_recall: 0.9340
Epoch 20/50
312/312 [==============================] - 1s 4ms/step - loss: 0.4749 - precision: 0.1476 - recall: 0.9007 - val_loss: 0.5594 - val_precision: 0.2131 - val_recall: 0.9340
Epoch 21/50
312/312 [==============================] - 1s 4ms/step - loss: 0.4785 - precision: 0.1461 - recall: 0.8951 - val_loss: 0.5551 - val_precision: 0.2165 - val_recall: 0.9323
Epoch 22/50
312/312 [==============================] - 2s 5ms/step - loss: 0.4838 - precision: 0.1461 - recall: 0.8876 - val_loss: 0.5556 - val_precision: 0.2166 - val_recall: 0.9323
Epoch 23/50
312/312 [==============================] - 2s 6ms/step - loss: 0.4701 - precision: 0.1520 - recall: 0.9007 - val_loss: 0.5415 - val_precision: 0.2240 - val_recall: 0.9323
Epoch 24/50
312/312 [==============================] - 2s 5ms/step - loss: 0.4674 - precision: 0.1561 - recall: 0.8989 - val_loss: 0.5277 - val_precision: 0.2348 - val_recall: 0.9323
Epoch 25/50
312/312 [==============================] - 1s 5ms/step - loss: 0.4670 - precision: 0.1575 - recall: 0.8970 - val_loss: 0.5252 - val_precision: 0.2372 - val_recall: 0.9323
Epoch 26/50
312/312 [==============================] - 3s 9ms/step - loss: 0.4608 - precision: 0.1598 - recall: 0.8933 - val_loss: 0.5168 - val_precision: 0.2439 - val_recall: 0.9306
Epoch 27/50
312/312 [==============================] - 6s 18ms/step - loss: 0.4552 - precision: 0.1629 - recall: 0.9007 - val_loss: 0.5190 - val_precision: 0.2399 - val_recall: 0.9306
Epoch 28/50
312/312 [==============================] - 3s 10ms/step - loss: 0.4572 - precision: 0.1678 - recall: 0.9157 - val_loss: 0.5115 - val_precision: 0.2467 - val_recall: 0.9306
Epoch 29/50
312/312 [==============================] - 2s 6ms/step - loss: 0.4579 - precision: 0.1622 - recall: 0.8839 - val_loss: 0.5045 - val_precision: 0.2530 - val_recall: 0.9288
Epoch 30/50
312/312 [==============================] - 1s 4ms/step - loss: 0.4434 - precision: 0.1727 - recall: 0.9139 - val_loss: 0.4904 - val_precision: 0.2659 - val_recall: 0.9271
Epoch 31/50
312/312 [==============================] - 1s 5ms/step - loss: 0.4440 - precision: 0.1758 - recall: 0.9045 - val_loss: 0.4861 - val_precision: 0.2696 - val_recall: 0.9271
Epoch 32/50
312/312 [==============================] - 1s 4ms/step - loss: 0.4365 - precision: 0.1779 - recall: 0.9026 - val_loss: 0.4818 - val_precision: 0.2718 - val_recall: 0.9271
Epoch 33/50
312/312 [==============================] - 1s 4ms/step - loss: 0.4383 - precision: 0.1790 - recall: 0.8989 - val_loss: 0.4802 - val_precision: 0.2736 - val_recall: 0.9271
Epoch 34/50
312/312 [==============================] - 1s 5ms/step - loss: 0.4358 - precision: 0.1876 - recall: 0.8933 - val_loss: 0.4631 - val_precision: 0.2884 - val_recall: 0.9219
Epoch 35/50
312/312 [==============================] - 1s 5ms/step - loss: 0.4268 - precision: 0.1864 - recall: 0.9064 - val_loss: 0.4619 - val_precision: 0.2880 - val_recall: 0.9219
Epoch 36/50
312/312 [==============================] - 1s 4ms/step - loss: 0.4236 - precision: 0.1912 - recall: 0.9026 - val_loss: 0.4649 - val_precision: 0.2841 - val_recall: 0.9219
Epoch 37/50
312/312 [==============================] - 2s 6ms/step - loss: 0.4280 - precision: 0.1911 - recall: 0.8970 - val_loss: 0.4483 - val_precision: 0.3006 - val_recall: 0.9201
Epoch 38/50
312/312 [==============================] - 2s 6ms/step - loss: 0.4193 - precision: 0.2024 - recall: 0.8933 - val_loss: 0.4467 - val_precision: 0.3011 - val_recall: 0.9236
Epoch 39/50
312/312 [==============================] - 1s 4ms/step - loss: 0.4225 - precision: 0.1972 - recall: 0.8933 - val_loss: 0.4433 - val_precision: 0.3042 - val_recall: 0.9236
Epoch 40/50
312/312 [==============================] - 1s 5ms/step - loss: 0.4202 - precision: 0.2069 - recall: 0.9101 - val_loss: 0.4358 - val_precision: 0.3094 - val_recall: 0.9201
Epoch 41/50
312/312 [==============================] - 1s 4ms/step - loss: 0.4156 - precision: 0.2061 - recall: 0.8951 - val_loss: 0.4302 - val_precision: 0.3166 - val_recall: 0.9201
Epoch 42/50
312/312 [==============================] - 1s 5ms/step - loss: 0.4147 - precision: 0.2037 - recall: 0.8914 - val_loss: 0.4212 - val_precision: 0.3245 - val_recall: 0.9184
Epoch 43/50
312/312 [==============================] - 1s 5ms/step - loss: 0.4159 - precision: 0.2069 - recall: 0.8820 - val_loss: 0.4170 - val_precision: 0.3319 - val_recall: 0.9184
Epoch 44/50
312/312 [==============================] - 1s 4ms/step - loss: 0.4129 - precision: 0.2095 - recall: 0.8858 - val_loss: 0.4236 - val_precision: 0.3185 - val_recall: 0.9201
Epoch 45/50
312/312 [==============================] - 1s 5ms/step - loss: 0.4041 - precision: 0.2154 - recall: 0.8989 - val_loss: 0.4161 - val_precision: 0.3284 - val_recall: 0.9184
Epoch 46/50
312/312 [==============================] - 2s 7ms/step - loss: 0.4117 - precision: 0.2160 - recall: 0.8820 - val_loss: 0.4059 - val_precision: 0.3420 - val_recall: 0.9149
Epoch 47/50
312/312 [==============================] - 2s 6ms/step - loss: 0.4050 - precision: 0.2164 - recall: 0.8895 - val_loss: 0.4024 - val_precision: 0.3506 - val_recall: 0.9149
Epoch 48/50
312/312 [==============================] - 1s 4ms/step - loss: 0.3994 - precision: 0.2247 - recall: 0.8820 - val_loss: 0.3970 - val_precision: 0.3539 - val_recall: 0.9149
Epoch 49/50
312/312 [==============================] - 1s 5ms/step - loss: 0.3990 - precision: 0.2276 - recall: 0.8876 - val_loss: 0.3950 - val_precision: 0.3566 - val_recall: 0.9149
Epoch 50/50
312/312 [==============================] - 1s 5ms/step - loss: 0.4033 - precision: 0.2261 - recall: 0.8933 - val_loss: 0.3848 - val_precision: 0.3697 - val_recall: 0.9167
In [101]:
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [102]:
output = model_performance(model13, X_test, y_test)
print(output)
output_1 = model_performance(model13, X_train, y_train)
print(output_1)

156/156 [==============================] - 0s 1ms/step
   Accuracy    Recall  Precision  F1 Score
0  0.906394  0.906394   0.956138  0.923622
312/312 [==============================] - 0s 1ms/step
   Accuracy    Recall  Precision  F1 Score
0  0.906231  0.906231   0.959188  0.924512
In [103]:
results.loc[13] = ['2','80,160','relu,relu',epochs,batch_size,'yes','he/glorot/randnorm','0.3',"Adam_LR(le-5)/L1,L2",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
                   ,output['Precision'][0],output['Recall'][0]]

results
Out[103]:
hidden_layers neurons_HL activation_ftn_HL epochs batch_size Batch_Norm Weight_Ini dropout optimizer time(secs) Train_loss Validation_loss Train_Precision Validation_Precision Train_Recall Validation_Recall Test_Precision Test_Recall
0 - - - 10 9982 - - - SGD 1.234703 2.031547 1.695963 0.054343 0.056860 0.219101 0.189236 0.890782 0.764682
1 - - - 25 9982 - - - SGD_LR(0.01) 3.018212 0.425082 0.430649 0.402844 0.445483 0.318352 0.248264 0.928051 0.940469
2 - - - 50 9982 - - - SGD-LR(0.01) 2.985321 0.434529 0.493189 0.566038 0.598456 0.280899 0.269097 0.942613 0.951694
3 - - - 25 32 - - - SGD-LR(0.01) 27.868973 2.379025 2.360701 0.022727 0.023452 0.046816 0.043403 0.888372 0.844257
4 1 80 relu 25 32 - - - SGD_LR(0.01) 41.567311 0.193805 0.144769 0.608861 0.707650 0.900749 0.899306 0.973761 0.970134
5 2 80,160 relu,relu 25 32 - - - SGD_LR(0.01) 41.894305 0.139635 0.115674 0.714076 0.767407 0.911985 0.899306 0.978480 0.977551
6 2 80,160 relu,relu 25 32 - - - Adam_LR(0.01) 42.051322 0.145135 0.133539 0.785032 0.840266 0.923221 0.876736 0.982125 0.981760
7 2 80,160 relu,relu 25 32 - - - Adam_LR(le-5) 31.487571 0.322984 0.313230 0.305324 0.337232 0.891386 0.901042 0.953981 0.898577
8 2 80,160 relu,relu 25 32 - - 0.3 Adam_LR(le-5) 42.194058 0.425188 0.440071 0.187967 0.218922 0.848315 0.895833 0.946604 0.818000
9 2 80,160 relu,relu 25 32 yes - 0.3 Adam_LR(le-5) 42.708833 0.450809 0.401692 0.145643 0.270876 0.867041 0.923611 0.951479 0.854279
10 2 80,160 relu,relu 25 32 yes he/glorot/randnorm 0.3 Adam_LR(le-5) 37.227174 0.436856 0.494443 0.144936 0.217604 0.895131 0.927083 0.948019 0.801764
11 2 80,160 relu,relu 25 32 yes he/glorot/randnorm 0.3 Adam_LR(le-5)/L1,L2 42.635221 0.467466 0.559652 0.150646 0.213456 0.895131 0.925347 0.946944 0.797354
12 2 80,160 relu,relu 25 32 yes he/glorot/randnorm 0.3 Adagrad_LR(0.01)/L1,L2 37.517705 0.256188 0.150753 0.528302 0.829618 0.891386 0.904514 0.981178 0.980758
13 2 80,160 relu,relu 50 32 yes he/glorot/randnorm 0.3 Adam_LR(le-5)/L1,L2 83.503170 0.403283 0.384788 0.226066 0.369748 0.893258 0.916667 0.956138 0.906394

Actionable Insights & Buisiness Recommendations¶

Actionable Insights:

  1. The model with best performance was Model13 with Recall metrics of importance as follows,

Testing - Recall - 0.906394

Training - Recall - 0.906231

  1. Configuration used for Model13 is as follows,
  • Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes, Output layer - Sigmoid Activaion Function
  • Adam Optimizer - learning_rate = 1e-5
  • Loss function - Binary Cross entropy
  • Epochs - 50 / Batch size - 32
  • Class weights taken into account
  • Drop out - 0.3 / Batch Normalization added
  • Weight Initialization done - He (HL1), Glorot (HL2), RandomNormal (Output)
  • Regularization - L1 and L2

  1. Started working with shallow Networks and then worked my way by increasing the epochs first and then batch size. Increasing the epoch got the smoothness to the metrics curve.

  2. Adam Optimizer proved to be a better choice compared to standard SGD and Adagrad. Adam with a slower learning rate worked better.

  3. Other techniques like Batch normalization, dropout of 30%, various weight initialization and L1,L2 regularization techniques helped a lot in improving and smoothening the gap between the Training and Testing Metrics.

Buisiness Recommendations:

  1. Features V7,V11,V15,V16,V21,V28,V34 have the most impact on determining whether a windmill will fail or not.¶
  2. The company should focus on improving the important features so they can reduce the number of failures and the amount of money spent on repairs and replacements.